#28414 closed defect (fixed)
p_group_cohomology does not build on Python3
| Reported by: | dimpase | Owned by: | |
|---|---|---|---|
| Priority: | major | Milestone: | sage-9.0 |
| Component: | packages: optional | Keywords: | py3 |
| Cc: | SimonKing | Merged in: | |
| Authors: | Simon King | Reviewers: | John Palmieri |
| Report Upstream: | N/A | Work issues: | |
| Branch: | 6c5dafc (Commits, GitHub, GitLab) | Commit: | |
| Dependencies: | #28444 | Stopgaps: |
GitHub link to the corresponding issue
Description (last modified by )
The aim of this ticket is to provide a new version of the optional p_group_cohomology spkg so that it builds and passes tests both with python-2 and python-3.
Hopefully the version number 3.3 (two times three) is appropriate.
Attachments (1)
Change History (167)
comment:2 Changed 3 years ago by
Simon, I have made some progress in converting this to Python 3. Let me know and I can email you some diff files.
comment:3 Changed 3 years ago by
Hi John, in fact Python-3-compatibility was one of the things that I considered for the next package version. So, if you already have a diff, I'd appreciate to see it!
comment:4 follow-ups: 5 36 Changed 3 years ago by
I just sent email to your Jena address. I can get the package to build, but it doesn't work yet. An illustration of one problem:
sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
#!/usr/bin/env sage-python23
WARNING: Files on disk have been moved or are not writeable.
> Will try to recover later.
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py:623: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
8gp3: Files on disk have been moved - trying to get things right
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-5-9bd7fbf5ad54> in <module>()
----> 1 load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj')
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in __call__(self, result)
244 self.start_displayhook()
245 self.write_output_prompt()
--> 246 format_dict, md_dict = self.compute_format_data(result)
247 self.update_user_ns(result)
248 self.fill_exec_result(result)
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/IPython/core/displayhook.py in compute_format_data(self, result)
148
149 """
--> 150 return self.shell.display_formatter.format(result)
151
152 # This can be set to True by the write_output_prompt method in a subclass
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/display/formatter.py in format(self, obj, include, exclude)
198 """
199 # First, use Sage rich output if there is any
--> 200 sage_format, sage_metadata = self.dm.displayhook(obj)
201 assert PLAIN_TEXT in sage_format, 'plain text is always present'
202 if not set(sage_format.keys()).issubset(self.default_mime()):
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in displayhook(self, obj)
809 return
810 self._backend.set_underscore_variable(obj)
--> 811 plain_text, rich_output = self._rich_output_formatter(obj, dict())
812 return self._backend.displayhook(plain_text, rich_output)
813
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/repl/rich_output/display_manager.py in _rich_output_formatter(self, obj, rich_repr_kwds)
621 rich_output = None
622 plain_text = None
--> 623 has_rich_repr = isinstance(obj, SageObject) and hasattr(obj, '_rich_repr_')
624 if has_rich_repr:
625 rich_output = self._call_rich_repr(obj, rich_repr_kwds)
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getattr__()
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
149
150 ## Load file by absolute filename
--> 151 with open(filename, 'rb') as fobj:
152 X = loads(fobj.read(), compress=compress)
153 try:
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
150 ## Load file by absolute filename
151 with open(filename, 'rb') as fobj:
--> 152 X = loads(fobj.read(), compress=compress)
153 try:
154 X._default_filename = os.path.abspath(filename)
/Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
967
968 unpickler = SageUnpickler(io.BytesIO(s))
--> 969 return unpickler.load()
970
971
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
comment:5 Changed 3 years ago by
Replying to jhpalmieri:
I just sent email to your Jena address. I can get the package to build, but it doesn't work yet.
Thank you very much, I received the patch.
An illustration of one problem:
sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj') /Users/palmieri/Desktop/Sage_stuff/sage_builds/PYTHON3/sage-8.9.beta8/src/bin/sage-ipython:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead ... UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
That probably is harmless. I.e., I should use "warning", and apparently in some unpickling method I use "str" where it should be "bytes".
comment:6 Changed 3 years ago by
Yes, and I would like to track down the str vs. bytes issue. It would be nice if I could get basic functionality like CohomologyRing(8,3) to work.
comment:7 follow-up: 8 Changed 3 years ago by
Changing logger.warn (or coho_logger.warn) to X.warning fixes the deprecation message. I don't know about the str/bytes issue.
comment:8 Changed 3 years ago by
Replying to jhpalmieri:
Changing
logger.warn(orcoho_logger.warn) toX.warningfixes the deprecation message. I don't know about the str/bytes issue.
I did change .warn( into .warning( everywhere (so, no need to include it in a diff).
Concerning str vs. byte: There is only one occurrence of cdef str, and this gets assigned to the output of <some singular object>.name(). Is that output really a string in python-3? If not, it should be changed to cdef bytes. However, so far the error in unpickling stuff looks mysterious to me, as the traceback doesn't mention where the problem actually occurs in my code.
comment:9 follow-up: 10 Changed 3 years ago by
Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?
comment:10 follow-up: 13 Changed 3 years ago by
Replying to jhpalmieri:
Could the problem be in the way the .sobj files were produced? What if you regenerate them using Python 3?
You mean Python 3 is severely backwards incompatible and cannot deal with old pickles? That would be a show stopper for Sage, I think.
Note that at some point in the past I inserted code to regenerate very old pickles and translate them automatically to a new format. However, I would certainly not like to regenerate several gigabytes of data in the web repository.
Note that the problem occurs together with a warning:
WARNING: Files on disk have been moved or are not writeable. > Will try to recover later. ... 8gp3: Files on disk have been moved - trying to get things right
Maybe that indicates where the problem comes from. So, let's recall how storing the modular cohomology ring of a prime power group is solved.
Very early in the package development I found that is is terribly slow to pack all data of a ring in a single file. It simply takes so long and requires so much memory (at least in some cases) that it is not feasible to automatically store intermediate results of a lengthy computation (which evidently is a reasonable thing to do).
My solution was to not store a ring in a single file but distribute the data in a folder. So, the actual .sobj pickle file only contains basic information but is not stand-alone. In particular, the p_group_cohomology spkg needs to know where the data folder is located, the .sobj file has to be in the data folder and also it needs write permission for otherwise it couldn't do a continuation of the cohomology ring's computation. What is stored in the web repository is a compressed tar file of the data folder (i.e., when the web repository is accessed, then the tar file is downloaded, unpacked, and then the cohomology ring is loaded from there).
By default, the data folder's location is composed from the location of the current workspace (which are global variables) and from the identifier of the group (which is stored in the .sobj file).
The warning mentioned above occurs when the data folder's default location cannot be accessed. And in that case, I do some trickery --- and perhaps that trickery doesn't work in Python 3? Namely:
- When an object is obtained from reading a pickle file, then Python (at least Python 2) stores the original file location as an attribute of the object. However, that attribute is only available after successful unpickling.
- Therefore, when initially the location of the data folder cannot be determined, an unitialised cohomology ring is returned.
- As soon as the uninitialised cohomology ring is touched (I think constructing its repr is enough), initialisation is triggered. Namely, recall that the location of the .sobj file is stored as an attribute of the uninitialised ring and that the .sobj file is supposed to be located in the data folder -- voilà, you can now determine the data folder and finish initialisation of the cohomology ring.
If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.
comment:11 follow-up: 12 Changed 3 years ago by
Question: How to test whether something has a type that qualifies it as a filename? Is isinstance(self._property_dict['_need_new_root'],basestring) ok in Python-3?
comment:12 follow-up: 31 Changed 3 years ago by
Replying to SimonKing:
Question: How to test whether something has a type that qualifies it as a filename? Is
isinstance(self._property_dict['_need_new_root'],basestring)ok in Python-3?
Here is what I found in the "internetz" concerning how to test if something is a string in a way that works in python 2 and 3:
import six
if isinstance(obj, six.string_types):
print('obj is a string!')
So, I think I should fix this.
comment:13 follow-up: 88 Changed 3 years ago by
Replying to SimonKing:
If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.
Hooray:
# in Sage-with-python-2 sage: P = CombinatorialFreeModule(ZZ,'x') sage: save(P,'../../pickle')
# BOTH with python-2 and python-3
sage: B = load('../../pickle.sobj')
sage: B._default_filename
'/home/king/Sage/pickle.sobj'
So, the old trick should still work.
comment:14 Changed 3 years ago by
Currently, I get the following behaviour:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_workspace(tmp_dir()) sage: H = CohomologyRing(8,3,from_scratch=True,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/' Computing basic setup for Small Group number 1 of order 2 Computing basic setup for Small Group number 2 of order 4 Computing basic setup for Small Group number 3 of order 8 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-3-d1c5c11acd7a> in <module>() ----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug') /home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False -> 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: -> 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__() TypeError: expected bytes, str found
and
sage: H = CohomologyRing(8,3,options='debug')
Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj
Creating symbolic links from /home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3
Group data are rooted at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/'
The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/26693/dir_f6_f2ei3/8gp3/dat/State.sobj'
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
1154 coho_options['@use_this_root@'] = root_workspace
-> 1155 OUT = load(os.path.join(root_workspace,file_name)) # realpath here?
1156 if '@use_this_root@' in coho_options:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
150 ## Load file by absolute filename
--> 151 with open(filename, 'rb') as fobj:
152 X = loads(fobj.read(), compress=compress)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
151 with open(filename, 'rb') as fobj:
--> 152 X = loads(fobj.read(), compress=compress)
153 try:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
968 unpickler = SageUnpickler(io.BytesIO(s))
--> 969 return unpickler.load()
970
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO_unpickle()
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
150 ## Load file by absolute filename
--> 151 with open(filename, 'rb') as fobj:
152 X = loads(fobj.read(), compress=compress)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
151 with open(filename, 'rb') as fobj:
--> 152 X = loads(fobj.read(), compress=compress)
153 try:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
968 unpickler = SageUnpickler(io.BytesIO(s))
--> 969 return unpickler.load()
970
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
During handling of the above exception, another exception occurred:
OSError Traceback (most recent call last)
<ipython-input-4-927128be23f4> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),options='debug')
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
1707 if q < 128:
1708 extras['websource'] = False
-> 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
1710 return OUT
1711
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
1159 if '@use_this_root@' in coho_options:
1160 del coho_options['@use_this_root@']
-> 1161 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_local_sources,file_name), msg))
1162 ## 4. Search web repository
1163 elif kwds.get('websource')!=False and (not from_scratch):
OSError: Saved data at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj are not readable: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
So, that's something I can now try to debug.
comment:15 follow-up: 16 Changed 3 years ago by
in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3
comment:16 Changed 3 years ago by
Replying to chapoton:
in case you need them, sage has its own compatibility functions "bytes_to_str" and "str_to_bytes", to be found in sage.cpython.string and working in both py2 and py3
Thank you! I don't know yet whether or not I'll need them, but certainly it is good to know.
comment:17 Changed 3 years ago by
I found that this fails in RESL.__init__:
tmprstem = os.path.join(res_folder,rstem)
tmpgstem = os.path.join(gps_folder,gstem)
self.Data = newResolWithGroupLoaded(tmprstem,tmpgstem,1)
Apparently tmprstem or tmpgstem are strings, but the C function newResolWithGroupLoaded expects char * --- is there no automatic conversion from py3-string to char*?
comment:18 follow-up: 19 Changed 3 years ago by
Maybe try
self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)
..with no guarantee..
comment:19 Changed 3 years ago by
Replying to chapoton:
Maybe try
self.Data = newResolWithGroupLoaded(str_to_bytes(tmprstem),str_to_bytes(tmpgstem),1)..with no guarantee..
That is likely to work. Another potential solution would be to generally work with bytes. I.e., let self.gstem be a bytes and then do os.path.join(self.res_folder, b'A'+self.gstem+b'.sobj')) etc.
What would be the preferred way?
comment:20 follow-up: 21 Changed 3 years ago by
beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes
EDIT:
indeed:
sage: os.path.join('home', b'jojo')
...
TypeError: Can't mix strings and bytes in path components
comment:21 Changed 3 years ago by
Replying to chapoton:
beware that you cannot concatenate bytes and str, so everything must become bytes. Not the way to go in my opinion.. moreover, not sure that os.* can handle bytes
EDIT:
indeed:
sage: os.path.join('home', b'jojo') ... TypeError: Can't mix strings and bytes in path components
This only shows that it cannot mix both types. In fact, if both are bytes then things work:
sage: os.path.join(b'home', b'foo') b'home/foo'
Probably it would be a long and tedious work to change everything to bytes. In principle I would be willing to do that work, but if there is a different solution involving less work and only a slight overhead, I'd prefer that.
Currently, using str_to_bytes when passing a string to a C function does seem to work.
comment:22 Changed 3 years ago by
WTF? I just see that I have the following argument types of COCH.__init__:
def __init__(self, PARENT, int n, char *Nick, L, ydeg=None, rdeg=None, is_polyrep=False)
I suppose Nick shouldn't be char* but (if anything) str. Or better not type it at all.
comment:23 Changed 3 years ago by
I think I should also fix the following compiler warning (there are several similar):
pGroupCohomology/cohomology.c: In function ‘__pyx_pf_16pGroupCohomology_10cohomology_12explore_one_parameter’:
pGroupCohomology/cohomology.c:17678:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
for (__pyx_t_7 = 0; __pyx_t_7 < __pyx_t_16; __pyx_t_7+=1) {
comment:24 Changed 3 years ago by
I guesss getting the following error is a progress, although it seems a bit strange to me:
sage: H = CohomologyRing(8,3,from_scratch=True,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
Computing basic setup for Small Group number 1 of order 2
Computing basic setup for Small Group number 2 of order 4
Computing basic setup for Small Group number 3 of order 8
> export action matrices
Initialising maximal p-elementary abelian subgroups
Inserting SmallGroup(2,1) as a subgroup
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/31253/dir_88iattmw/'
> export action matrices
Unable to save basic ring setup
Traceback (most recent call last):
File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)
obj.save(filename=filename, compress=compress, **kwargs)
File "sage/structure/sage_object.pyx", line 424, in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)
filename = _base_save(self, filename, compress=compress)
File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
with open(filename, 'wb') as fobj:
File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
fobj.write(_base_dumps(obj, compress=compress))
File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
gherkin = SagePickler.dumps(obj)
File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
pickler.dump(obj)
File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
save (obj, path)
File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
_base_save(obj, filename, compress=compress)
File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
with open(filename, 'wb') as fobj:
File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
fobj.write(_base_dumps(obj, compress=compress))
File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
gherkin = SagePickler.dumps(obj)
File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
pickler.dump(obj)
TypeError: can't pickle dict_items objects
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "sage/misc/persist.pyx", line 240, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)
obj.save(filename=filename, compress=compress, **kwargs)
AttributeError: 'tuple' object has no attribute 'save'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py", line 1254, in _get_p_group_from_scratch
safe_save(OUT, OUT.autosave_name())
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
save (obj, path)
File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
_base_save(obj, filename, compress=compress)
File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
with open(filename, 'wb') as fobj:
File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
fobj.write(_base_dumps(obj, compress=compress))
File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
gherkin = SagePickler.dumps(obj)
File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
pickler.dump(obj)
File "pGroupCohomology/cohomology.pyx", line 3464, in pGroupCohomology.cohomology.COHO.__reduce__
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py", line 85, in safe_save
save (obj, path)
File "sage/misc/persist.pyx", line 242, in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)
_base_save(obj, filename, compress=compress)
File "sage/misc/persist.pyx", line 179, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)
with open(filename, 'wb') as fobj:
File "sage/misc/persist.pyx", line 180, in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)
fobj.write(_base_dumps(obj, compress=compress))
File "sage/misc/persist.pyx", line 257, in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)
gherkin = SagePickler.dumps(obj)
File "sage/misc/persist.pyx", line 836, in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)
pickler.dump(obj)
TypeError: can't pickle dict_items objects
Computing next term
> rk P_02 = 1
We have to choose 1 new generator in degree 1
> There is 1 Duflot regular generator in degree 1
Summary: 0 relations and 1 generators in degree 1
Ring approximation computed out to degree 1!
Storing approximation data
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
239 try:
--> 240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
239 try:
--> 240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
AttributeError: 'dict_items' object has no attribute 'save'
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3636)()
239 try:
--> 240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/structure/sage_object.pyx in sage.structure.sage_object.SageObject.save (build/cythonized/sage/structure/sage_object.c:3328)()
423
--> 424 filename = _base_save(self, filename, compress=compress)
425
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
178
--> 179 with open(filename, 'wb') as fobj:
180 fobj.write(_base_dumps(obj, compress=compress))
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
179 with open(filename, 'wb') as fobj:
--> 180 fobj.write(_base_dumps(obj, compress=compress))
181
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
256
--> 257 gherkin = SagePickler.dumps(obj)
258
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
835 pickler = cls(buf, **kwargs)
--> 836 pickler.dump(obj)
837 return buf.getvalue()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
84 os.unlink(path)
---> 85 save (obj, path)
86
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
241 except (AttributeError, RuntimeError, TypeError):
--> 242 _base_save(obj, filename, compress=compress)
243 else:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
178
--> 179 with open(filename, 'wb') as fobj:
180 fobj.write(_base_dumps(obj, compress=compress))
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
179 with open(filename, 'wb') as fobj:
--> 180 fobj.write(_base_dumps(obj, compress=compress))
181
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
256
--> 257 gherkin = SagePickler.dumps(obj)
258
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
835 pickler = cls(buf, **kwargs)
--> 836 pickler.dump(obj)
837 return buf.getvalue()
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
84 os.unlink(path)
---> 85 save (obj, path)
86
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
241 except (AttributeError, RuntimeError, TypeError):
--> 242 _base_save(obj, filename, compress=compress)
243 else:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
178
--> 179 with open(filename, 'wb') as fobj:
180 fobj.write(_base_dumps(obj, compress=compress))
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
179 with open(filename, 'wb') as fobj:
--> 180 fobj.write(_base_dumps(obj, compress=compress))
181
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
256
--> 257 gherkin = SagePickler.dumps(obj)
258
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
835 pickler = cls(buf, **kwargs)
--> 836 pickler.dump(obj)
837 return buf.getvalue()
TypeError: can't pickle dict_items objects
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
239 try:
--> 240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
AttributeError: 'pGroupCohomology.resolution.RESL' object has no attribute 'save'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3627)()
239 try:
--> 240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
AttributeError: 'dict_items' object has no attribute 'save'
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
<ipython-input-3-d1c5c11acd7a> in <module>()
----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug')
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds)
1707 if q < 128:
1708 extras['websource'] = False
-> 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
1710 return OUT
1711
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds)
1246 OUT = COHO(gap(KEY[0]), **extras)
1247 else:
-> 1248 OUT = COHO(KEY[0],KEY[1], **extras)
1249 _gap_reset_random_seed()
1250 try:
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.make()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.next()
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
83 if os.path.islink(path):
84 os.unlink(path)
---> 85 save (obj, path)
86
87 ###################################
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
--> 242 _base_save(obj, filename, compress=compress)
243 else:
244 # Saving an object to an image file.
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
177 filename = _normalize_filename(filename)
178
--> 179 with open(filename, 'wb') as fobj:
180 fobj.write(_base_dumps(obj, compress=compress))
181
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
178
179 with open(filename, 'wb') as fobj:
--> 180 fobj.write(_base_dumps(obj, compress=compress))
181
182 return filename
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
255 """
256
--> 257 gherkin = SagePickler.dumps(obj)
258
259 if compress:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
834 buf = io.BytesIO()
835 pickler = cls(buf, **kwargs)
--> 836 pickler.dump(obj)
837 return buf.getvalue()
838
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__reduce__()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__getstate__()
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
83 if os.path.islink(path):
84 os.unlink(path)
---> 85 save (obj, path)
86
87 ###################################
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
--> 242 _base_save(obj, filename, compress=compress)
243 else:
244 # Saving an object to an image file.
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
177 filename = _normalize_filename(filename)
178
--> 179 with open(filename, 'wb') as fobj:
180 fobj.write(_base_dumps(obj, compress=compress))
181
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
178
179 with open(filename, 'wb') as fobj:
--> 180 fobj.write(_base_dumps(obj, compress=compress))
181
182 return filename
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
255 """
256
--> 257 gherkin = SagePickler.dumps(obj)
258
259 if compress:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
834 buf = io.BytesIO()
835 pickler = cls(buf, **kwargs)
--> 836 pickler.dump(obj)
837 return buf.getvalue()
838
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__reduce__()
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.exportLifts()
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.LIFTcontainer.export()
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/auxiliaries.py in safe_save(obj, path)
83 if os.path.islink(path):
84 os.unlink(path)
---> 85 save (obj, path)
86
87 ###################################
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.save (build/cythonized/sage/misc/persist.c:3694)()
240 obj.save(filename=filename, compress=compress, **kwargs)
241 except (AttributeError, RuntimeError, TypeError):
--> 242 _base_save(obj, filename, compress=compress)
243 else:
244 # Saving an object to an image file.
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3318)()
177 filename = _normalize_filename(filename)
178
--> 179 with open(filename, 'wb') as fobj:
180 fobj.write(_base_dumps(obj, compress=compress))
181
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_save (build/cythonized/sage/misc/persist.c:3253)()
178
179 with open(filename, 'wb') as fobj:
--> 180 fobj.write(_base_dumps(obj, compress=compress))
181
182 return filename
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._base_dumps (build/cythonized/sage/misc/persist.c:3898)()
255 """
256
--> 257 gherkin = SagePickler.dumps(obj)
258
259 if compress:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SagePickler.dumps (build/cythonized/sage/misc/persist.c:6613)()
834 buf = io.BytesIO()
835 pickler = cls(buf, **kwargs)
--> 836 pickler.dump(obj)
837 return buf.getvalue()
838
TypeError: can't pickle dict_items objects
What are dict_items, and why can't they be pickled?
comment:25 follow-up: 26 Changed 3 years ago by
If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).
You may need to wrap with list(d.items()) for pickling.
comment:26 follow-ups: 28 32 Changed 3 years ago by
Replying to chapoton:
If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).
You may need to wrap with list(d.items()) for pickling.
Wouldn't that considerably slow things down, at least in Python-2? After all, d.items() is a list, and list(...) applied to a list creates a (shallow) copy of that list.
comment:27 follow-up: 29 Changed 3 years ago by
There is a considerable progress now (after putting list around .items()):
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_workspace(tmp_dir()) sage: H = CohomologyRing(8,3,from_scratch=True,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/' Computing basic setup for Small Group number 1 of order 2 Computing basic setup for Small Group number 2 of order 4 Computing basic setup for Small Group number 3 of order 8 > export action matrices Initialising maximal p-elementary abelian subgroups Inserting SmallGroup(2,1) as a subgroup We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/' > export action matrices Computing next term > rk P_02 = 1 We have to choose 1 new generator in degree 1 > There is 1 Duflot regular generator in degree 1 Summary: 0 relations and 1 generators in degree 1 Ring approximation computed out to degree 1! Storing approximation data Determine degree 2 standard monomials We got 1 standard monomials Monomial 0 > Candidate: c_1_0^2 > Express monomial as a Cochain Compute c_1_0*c_1_0 Compose chain maps R_2 -> R_1 -> R_0 Lift with Urbild Groebner basis in degree 1 load Urbild Groebner basis Decomposable cochain found There is no new generator in degree 2 Summary: 0 relations and 0 generators in degree 2 Ring approximation computed out to degree 2! Storing approximation data Inserting SmallGroup(4,2) as a subgroup We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/2449/dir_dxt9xtpq/' > export action matrices --------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-3-d1c5c11acd7a> in <module>() ----> 1 H = CohomologyRing(Integer(8),Integer(3),from_scratch=True,options='debug') /home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False -> 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: -> 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InitSubgroups() pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.InsertSubgroup() /home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in __call__(self, *args, **kwds) 1707 if q < 128: 1708 extras['websource'] = False -> 1709 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1710 return OUT 1711 /home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/factory.py in _get_p_group_from_scratch(self, KEY, q, GStem, GroupName, **kwds) 1246 OUT = COHO(gap(KEY[0]), **extras) 1247 else: -> 1248 OUT = COHO(KEY[0],KEY[1], **extras) 1249 _gap_reset_random_seed() 1250 try: pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__() KeyError: 'autoliftElAb'
The good news is that for the first time cohomology rings were computed in py-3 (namely cohomology rings of elementary abelian groups).
comment:28 Changed 3 years ago by
Replying to SimonKing:
Replying to chapoton:
If d is a dict, then d.items() is no longer a list, but a view (some special kind of iterable).
You may need to wrap with list(d.items()) for pickling.
Wouldn't that considerably slow things down, at least in Python-2? After all,
d.items()is a list, andlist(...)applied to a list creates a (shallow) copy of that list.
You could use a try ... except block to deal with the Python 2 case quickly. Sage will soon (I hope) default to Python 3, so I don't think this is worth it, though.
comment:29 Changed 3 years ago by
Replying to SimonKing:
There is a considerable progress now (after putting
listaround.items()): ...KeyError: 'autoliftElAb'
Apparently I forgot to create a shallow copy in some places in which I temporarily stored the global options -- it doesn't work to recreate a dict from its items in Py-3, and that's why I got the key error above:
sage: D = {1:2}
sage: I = D.items()
sage: D.clear()
sage: D.update(I)
sage: D
{}
In Py-2, one would get
sage: D
{1: 2}
comment:30 Changed 3 years ago by
Hooray!
I am now in a position to completely compute CohomologyRing(8,3) from scratch. Unfortunately reading old data doesn't work yet, but certainly it is a progress :-)
comment:31 follow-up: 33 Changed 3 years ago by
Replying to SimonKing:
import six
Please don't do this in Cython files. There is almost never a reason to use six in Cython since Cython handles most of the Python 2/3 incompatibilities for you.
comment:32 follow-up: 34 Changed 3 years ago by
comment:33 follow-up: 35 Changed 3 years ago by
Replying to jdemeyer:
Replying to SimonKing:
import sixPlease don't do this in Cython files. There is almost never a reason to use
sixin Cython since Cython handles most of the Python 2/3 incompatibilities for you.
I am not dealing with incompatibilities in that case. I import six in order to be able to safely test for string types. How to, if I don't use six?
comment:34 Changed 3 years ago by
Replying to jdemeyer:
Use
list(d.iteritems())in Cython.
Would this also work fast if Cython doesn't know that d is a dict? Should I do
list(<dict>(self._D).iteritems()), for instance?
comment:35 Changed 3 years ago by
Replying to SimonKing:
I import six in order to be able to safely test for string types. How to, if I don't use six?
First of all, you should check whether you really need to test for "string types". Can you use duck typing instead? If for some reason, duck typing is hard, then it depends what you mean precisely with "string types".
Since you were mentioning filenames: isinstance(s, (bytes, unicode)) are valid types for filenames both in Python 2 and Python 3. Note that Python 3 also has https://docs.python.org/3/library/pathlib.html, a custom class to deal specifically with filenames.
comment:36 Changed 3 years ago by
Replying to jhpalmieri:
I can get the package to build, but it doesn't work yet. An illustration of one problem:
sage: load('/Users/palmieri/.sage/pGroupCohomology/db/8gp3/H8gp3.sobj') ... UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
I tried to track down that problem, and currently it seems to me that it is due to a very bad backwards incompatibility in Python. You can reproduce it as follows:
- Download attachment:State.sobj
- Install the optional meataxe spkg (this may involve
sage -b, so that some optional module in the Sage library is built) - Define some unpickle helper:
sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle sage: class unpickle_old_mtx: ....: def __call__(self, *args, **kwds): ....: return mtx_unpickle(*args, **kwds) ....: sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
Now, in Sage-with-Py-2, loading the file works:
sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
sage: X
([[1,
[(2, 1),
[
[1 0 0 0 0 0 0 0]
[0 0 0 1 1 1 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg1_2']
]]],
[2,
[(4, 2),
[
[1 0 0 0 0 0 0 0]
[0 0 0 1 1 1 1 1]
[0 1 0 0 0 0 0 0]
[0 0 0 0 0 1 0 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg2_2']
]]],
[3,
[(4, 2),
[
[1 0 0 0 0 0 0 0]
[0 0 0 1 1 1 1 1]
[0 0 1 0 0 0 0 0]
[0 0 0 0 0 0 1 1], [[1 0 0 0 0 0 0 0], '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_1', '/home/king/SPKG/database/8gp3/sgp/8gp3sg3_2']
]]]],
[2, 1, 1],
1,
1,
'8gp3/',
['b_1_0*b_1_1'],
'8gp3/dat/',
[((4, 2), '/home/king/.sage/pGroupCohomology/db/4gp2/H4gp2'),
((2, 1), '/home/king/.sage/pGroupCohomology/db/2gp1/H2gp1')],
'8gp3/dat/',
[],
'8gp3/sgp/',
2,
[2, 3],
[2, 2],
2,
2,
['b_1_0*b_1_1'],
[[2, 'c_2_2', [0 0 1], 1, 0],
[1, 'b_1_0', [1 0], 0, 0],
[1, 'b_1_1', [0 1], 0, 0]],
[[0, [['1', ['1']]]],
[1, [['b_1_0', ['b_1_0']], ['b_1_1', ['b_1_1']]]],
[2,
[['c_2_2', ['c_2_2']],
['b_1_0', ['b_1_0^2']],
['b_1_1', ['b_1_0*b_1_1', 'b_1_1^2']]]]],
{},
0.24,
True,
'Monomials',
2,
None,
'',
0,
'8gp3',
'8gp3/dat/R8gp3.sobj',
1,
['0'],
['b_1_0 + b_1_1'],
None,
[('_max_module_deg', 0),
('sgpDickson', [((4, 2), [[1, 'c_1_1', [0 1]]])]),
('useFactorization', True),
('useElimination', False),
('GroupDescr', 'Dihedral group of order 8'),
('_key', ((8, 3), '/home/king/SPKG/database/8gp3/dat/State')),
('auto', 4),
('Restriction_2', ['c_1_0*c_1_1+c_1_0^2', 'c_1_1', '0']),
('GroupName', 'D8'),
('DicksonExp', 3),
('_parameters_do_exist', True),
('KeepBases', None),
('completeGroebner', True),
('_SymondsTestdata', ['c_2_2', 'b_1_1', 'b_1_0']),
('_parameters_for_criterion', ['c_2_2', 'b_1_1', 'b_1_0']),
('root', '/home/king/SPKG/database/'),
('Restriction_3', ['c_1_0*c_1_1+c_1_0^2', '0', 'c_1_1']),
('_method', 'Symonds')],
[(('dependent_parameters',), [['c_2_2', 'b_1_1', 'b_1_0'], 2])])
However, Sage-with-Py-3 yields the error found by John:
sage: X = load('/home/king/Projekte/coho/tests/State.sobj')
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-4-74e7e91f9969> in <module>()
----> 1 X = load('/home/king/Projekte/coho/tests/State.sobj')
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2824)()
149
150 ## Load file by absolute filename
--> 151 with open(filename, 'rb') as fobj:
152 X = loads(fobj.read(), compress=compress)
153 try:
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2774)()
150 ## Load file by absolute filename
151 with open(filename, 'rb') as fobj:
--> 152 X = loads(fobj.read(), compress=compress)
153 try:
154 X._default_filename = os.path.abspath(filename)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
967
968 unpickler = SageUnpickler(io.BytesIO(s))
--> 969 return unpickler.load()
970
971
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
And the point is: Python-3 even is unable to read the pickle file as a string! In Python-2, reading the file is no problem at all.
king@klap:~$ ~/Sage/git/py3/sage -python
Python 3.7.3 (default, Aug 27 2019, 23:22:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/king/Sage/git/py3/local/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 1: invalid start byte
>>>
king@klap:~$ ~/Sage/git/sage/sage --python
Python 2.7.15 (default, Jul 26 2019, 11:49:43)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('/home/king/Projekte/coho/tests/State.sobj')
>>> f.read()
'x\x9c\x95U\x8bZ\x1bE\x14&!\x91\x92J\x8b\x82V\xbc\x94x\xedXl\x96\x04\xaah\x95B\x0b\x05]\xc1\x16\x18\xad\x97\xe9:\xd9\x1dv\xb7d3{v7\nU,^\xd8B\xbd\xd6\xfb\xfd\xae\xcf\xd0\xa7\xf0E|\t\xcfl\xb2\x08\xf5\xab_I\xbe\x9d\xefd\xe6?\xe7\xfc\xe7?g6kY\xc2 \x83O\x96\xe8\x19\x06\xed\xc4\x0c\xb9-J\x81[\xb7\xc3\x92[\x8f\x84-\x82\x82\xc7\x97\x84\x91\xfe\x80\x1c\xcdT\xd6\xe7 \xef\xa0QF\xe3\xa6\x18:\x18\xec!\xc4\xf4\xa7\x02\xd9\xf0OJGz\xb2&\xed\x95\x92\x17-\x17f\x16\xce\x1a\x8d\xba\xef\x9aK5a\x985\x1e\x86\x05\xe8\x94PX\x85\xbd\xf4\x80a\x84|Q\x18\x8b2H1\x98\xd80\xe0\xe6\x8d\xb0J\xf4,~\xf7\xd0\xecZ\xffF4\x07]\x0c\xf6\x11\xa7\xa0v3\xb8\x9bY\xdb\xc4\xcd\xfdt@\xc3lB[B?m\xfe\xb4>\xa5Y<\xe2U\x1e\nm\xc4\xf6\x87\xb4\xd0\xf6\x13#\xb4\xcbF\x19\xbaw\x03\xaf\xc0-\x02?\x0cn\xc5\x9c\x0cz\x88*x\x18\x0b\xeeuZ\x12\xdc\x16\xc3\xed\x0c\x0e\xb4X\xe5\x90Un\xad\x7f,\xaf\xd8\xde\xc1\xa0\xef?l\xef\xbc\xf1\xf4\x15d{\xd7n\xe0\x15\xb8\xbb\xc9\xf6\x1e\xa2\xb738\xb8\xc5\xb6?e[\x8c\xe1^\x06\xf7\xedd[lWl\xefg\xf0\x00qvAo\x08\xe9=\xb8\x1bx\x05\x1e\x12M~\x87\x9a\x9adD:?$5\x1e\xa6\xf9\xc4\x8f\xc1a\xba\xb7j\x94\x8d\xc1\xc3j-\xc3\x00\xa7\x9d\xc9\t\xc6\xc7\xd3G\x88\xd3\xeb\xa0\xf6G\xa8\xb6\x8d@I\r\xaev\xed\x08jVU\x1b\xb6\xfd\x8a6\xad\xd6\x18JN\xde\xc1\x81\xd5n\xd0\xb5b\xfbemZ\xad1\x0c\x8a\xed,Z\xb6\xaaQMG\x99\xa42W\x941\x84\xc6\x10\x16;\xbc\xb5}45\x1e\x15\xceQ\xe5\xf1\xd8\xce\x1aG8\x83\xc7\xf1\x1e>\x81\xf2\xd0\xbci\xa8\x8e\x1eK\x07\xa8\x9df\x8aj\x80\x9eD\xbb\r\xc3>\x85\x97\x95\xe6\x13w\x18MA\xd9\xd6\x94\x1d\xd7\xdb\x12\xd0\xd8\x16\xa8\x0c\xe3\xdb@c\nt"\x01!\xea$\xe6\x9c z\x1b\x83I\x06\xa7\x08v\x82\xc1\x14\xae\\p<\x9dN\xde\nO#\xe6\x19\xe2\x8c2\xd0\xd3\xac\xcf\xaa\xd3\x19\xe2\x8c3\x98M\x93<\xc7\x93\x06\x9fN\xae\xcb\x19\xf4\x99#\xce1\x06\xf3i9\x0b\xca\x87\x924\xc4\xf3\x0c^\xa0\x1d\x89}\xae\x02g\xd5\xe1\x8b$\x8d\xf5\x12\x83\x97\xc9N\x85^i\x82\xcb\x08f\xc9,\xad\xc2\xb9\xa9\xe3\x7f]=\xf3\xf7\xfa\xc1\xab\x1b\xb4sF\xd6\xa5\xe7\xf2Z\x08F\xaa\xf5\xab\xb3\x14\xcb\xa49\xd5*\xe0\xb4g\xab}s\xca*\x85\xb2z^\xd5W\xa5\x99A\x94\xdf\xa4]I\xbe\xe2@\xb1\x99\xd1\xe2\xb3\x0c\x04\xa1\xfb\r\x8f/\x1b\x9e\xb4\x1a\xf8\n\xb3\x84\r\x8b*\xc1 &\xb0cph\x01\x87`\x02\xdf]\xa1\xac\x83\xcb\xe0\xbcs\x84\xc1\x12\x83Z\xa2\xbf\x99D\xf2\xae\xd5\xbf.x\x0c\x12\x1f\x9fv7Bq\x8a\x9b\x91\x0c\xdc\x0b<r1\x08l\xc4\x10\xd0}\xb8?Ys=\xb7\xde\xdc\r7c\x88h!\x99\xd2\t\x11\x9a\x014h\xdf\x84\xeb\x08+\xe0\xb5\xa2\xad\xb6\x8br\xb1(\x03K\x04\xc5\x11x-\x86\xd7i\xceX\x12+\xb0\xac\xe8\x8e \xdd\x95t6/\xc4\xf0\x06=\xf4\xff\xd7W\xe94\x1f\xf1H\xc4\xf0f\x0c\xab4\xc7\x1b\x91\x84\xb7\xf4\\\x0c\x17i\xd7\x9c\x08\xa3\xc05\x153l\xed\x1a\x83\xb7\t\xed1\x93v%%\x0f\x98\xad\xc6\xbe\x93j\xf0.J\x86\xb1\xde\xa3\x9dI\t\xb3\xdc\x13\xb0N\xb3\x13#\x10\xc7p\x89\x16Z\x12N.\xfb\xb0\xa1\xb7\xc7\xb0I{\r\x9f\x07\x08\x8bD\x10\x1a\x964\xc4\xb2\x1bFp\x19\xd5y\x9fv\xeaB\xf8\'\x90l\x08\x1f\xcc\xc6\xf0!\xed6\xa5\xe7\xd7\x10\x8b\xd1E\xb5.\x02\xf8\x08\x81\x1f\xd3nc~\xc5\x93u+\\@\xc6\xaaB\xf8\x84\xc1\x15\x9cLg\xdc\x19E>\x9f\xd2\xbe\xedi\xd4_\x91\x19\xb8h+\xcd?c\xf0\xf9\xbf\xd0/h.\x902\x82/i\xdfu\xa5\x83\xafb\xf8z\xa7<C\xf0\r\x83o\xaf#\xcfw\xa8J\xaa\xd0\xf7\x98\xe3\x07\xdaa \x15GZ\xf0#\xedhQ\x87\x9fb\xf8\x19\xef\xc7/\xb4\xd7\x12\xbe\xa8[\xa2\x1em#\r\xbf\xae\xc3o\x0c~\'\xce\x15=\x8b1\xfe\xe0\x11\xfcY\xfa\x07\x14\x1f\x95\xac'
comment:37 follow-up: 38 Changed 3 years ago by
This works fine in python3:
sage: with open('State.sobj', 'rb') as f:
....: f.read()
comment:38 Changed 3 years ago by
Replying to chapoton:
This works fine in python3:
sage: with open('State.sobj', 'rb') as f: ....: f.read()
Do I understand that it works because b means "binary"? But of course the real question is why Python 3 cannot unpickle from that file.
I tried to dig a bit deeper into Sage's load function (I guess the error is hidden there), and indeed load() uses with open(filename, 'rb') as fobj:. So, the unpickling problem is something else:
sage: from sage.matrix.matrix_gfpn_dense import mtx_unpickle
sage: filename = '/home/king/Projekte/coho/tests/State.sobj'
sage: compress = True
sage: class unpickle_old_mtx:
....: def __call__(self, *args, **kwds):
....: return mtx_unpickle(*args, **kwds)
....:
sage: register_unpickle_override('pGroupCohomology.mtx', 'MTX_unpickle_class', unpickle_old_mtx)
# The following is copied from Sage's load() function.
sage: ## Load file by absolute filename
....: with open(filename, 'rb') as fobj:
....: X = loads(fobj.read(), compress=compress)
....: try:
....: X._default_filename = os.path.abspath(filename)
....: except AttributeError:
....: pass
....:
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-9-654643a51ef6> in <module>()
1 ## Load file by absolute filename
2 with open(filename, 'rb') as fobj:
----> 3 X = loads(fobj.read(), compress=compress)
4 try:
5 X._default_filename = os.path.abspath(filename)
/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7270)()
967
968 unpickler = SageUnpickler(io.BytesIO(s))
--> 969 return unpickler.load()
970
971
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128)
So, now we see that the error occurs in SageUnpickler(io.BytesIO(x)).load().
comment:39 follow-up: 40 Changed 3 years ago by
I find it very difficult to track down. But the fact that one does see a mentioning of pGroupCohomology.mtx when one forgets to register_unpickle_override(...) seems to indicate that the error occurs when an mtx matrix is unpickled.
So, I am now trying to find if this is really the case or not.
comment:40 follow-up: 41 Changed 3 years ago by
Replying to SimonKing:
I find it very difficult to track down. But the fact that one does see a mentioning of
pGroupCohomology.mtxwhen one forgets toregister_unpickle_override(...)seems to indicate that the error occurs when an mtx matrix is unpickled.So, I am now trying to find if this is really the case or not.
It isn't. I inserted a print statement at the beginning of mtx_unpickle, but it isn't shown before the error triggered by load('/home/king/Projekte/coho/tests/State.sobj') is raised.
I'll open a ticket for that problem.
comment:41 Changed 3 years ago by
comment:42 Changed 3 years ago by
| Dependencies: | → #28444 |
|---|
comment:43 Changed 3 years ago by
It seems that #28444 fixes the problem for p_group_cohomology on Python-3. With the current unpublished code for the package, I get
$ ./sage ┌────────────────────────────────────────────────────────────────────┐ │ SageMath version 8.9.beta9, Release Date: 2019-09-02 │ │ Using Python 3.7.3. Type "help()" for help. │ └────────────────────────────────────────────────────────────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Warning: this is a prerelease version, and it may be unstable. ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.set_workspace(tmp_dir()) sage: H = CohomologyRing(8,3,options='debug') Local data found at /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3/H8gp3.sobj Creating symbolic links from /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3 to /home/king/Sage/git/py3/local/share/pGroupCohomology/8gp3 Get original root and GStem /home/king/SPKG/database 8gp3 now, StateFile is /home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj Try to get (8, 3) from cache Group data are rooted at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/' The state descriptor of the to-be-unpickled ring is expected to be provided at '/home/king/.sage/temp/klap/7894/dir_05fi7c3s/8gp3/dat/State.sobj' Setting state > export action matrices Differential reloaded > rk P_02 = 3 Differential reloaded > rk P_03 = 4 Import monomials Checking compatibility of SmallGroups library and stored cohomology ring sage: print(H) Cohomology ring of Dihedral group of order 8 with coefficients in GF(2) Computation complete Minimal list of generators: [c_2_2: 2-Cocycle in H^*(D8; GF(2)), b_1_0: 1-Cocycle in H^*(D8; GF(2)), b_1_1: 1-Cocycle in H^*(D8; GF(2))] Minimal list of algebraic relations: [b_1_0*b_1_1]
Hooray...
comment:44 Changed 3 years ago by
Strangely, while I can compute some cohomology rings, the following computation fails in py3:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: H = CohomologyRing(32,4) sage: H.make()
Reason: Automatic saving does not work, and apparently it is because H.__getstate__() returns Singular elements, which cannot be pickled. It seems I did take care of it, as H.__getstate__() in py2 only returns the class sage.interfaces.singular.SingularElement and not the instance, but for some reasons it doesn't in py3.
comment:45 Changed 3 years ago by
Question: Has the mechanism of tab completion changed between py2 and py3? I notice that tab completion on a cohomology ring works in py2, but doesn't work (no error, but no completion either) in py3.
comment:46 Changed 3 years ago by
tab completion is done by ipython, which should be upgraded (they've switched to Python 3 only, so we're using an old version, which in turn might lag in its Python 3 support).
I suggest not to worry about it now.
comment:47 Changed 3 years ago by
Aha! The culprit for pickle problem is pGroupCohomology.cohomology.pickle_gap_data, which inspite of its name also takes care of singular elements. In Py2:
sage: self = CohomologyRing(32,4)
sage: self.make()
sage: G = self._decorator_cache['_parameter_restrictions',frozenset({'c_2_2'}),('radical', True)]
sage: G
[[(polynomial ring, over a field, global ordering
// coefficients: ZZ/2
// number of vars : 2
// block 1 : ordering M
// : names c_1_0 c_1_1
// : weights 1 1
// : weights -1 0
// block 2 : ordering C,
<repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca690>) failed: AttributeError: 'NoneType' object has no attribute 'group'>,
<repr(<sage.interfaces.singular.SingularElement at 0x7ff4a0cca500>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)],
2]
sage: pickle_gap_data(G)
[[(<class 'sage.interfaces.singular.SingularElement'>,
<class 'sage.interfaces.singular.SingularElement'>,
<class 'sage.interfaces.singular.SingularElement'>)],
2]
In Py3:
sage: pickle_gap_data(G) [[(polynomial ring, over a field, global ordering // coefficients: ZZ/2 // number of vars : 2 // block 1 : ordering M // : names c_1_0 c_1_1 // : weights 1 1 // : weights -1 0 // block 2 : ordering C, <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86a20>) failed: AttributeError: 'NoneType' object has no attribute 'group'>, <repr(<sage.interfaces.singular.SingularElement at 0x7f6475b86dc8>) failed: AttributeError: 'NoneType' object has no attribute 'group'>)], 2]
comment:48 Changed 3 years ago by
Ouch. A nasty thing.
In pickle_gap_data, I test if the input is an iterable class, and if it is and if the class is a builtin, then the pickle_gap_data traverses the items of the input and creates from the individual results a modified copy of the input. In code:
try: I = iter(G) except: return G if isinstance(G,dict): return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items()) if getattr(type(G), '__module__', None) == '__builtin__': return type(G)(pickle_gap_data(X) for X in I) return G
The problem is that in Py3 we get
sage: getattr(list, '__module__', None) 'builtins'
Question: How to test in a py2/py3 invariant way whether a type is a builtin?
That asked, are there more iterable builtins than list, tuple, str/bytes and dict? I guess it is safer to just explicitly test for list and tuple. EDIT Like this:
if isinstance(G, dict): return dict((pickle_gap_data(k), pickle_gap_data(v)) for k,v in G.items()) if isinstance(G, (list, tuple)): return type(G)(pickle_gap_data(X) for X in G) return G
comment:49 follow-up: 50 Changed 3 years ago by
An iterable is an object with an __iter__ method or/and __getitem__ method defined.
So perhaps you can test for their presense. (These methods may be broken, but OK...)
comment:50 Changed 3 years ago by
Replying to dimpase:
An iterable is an object with an
__iter__method or/and__getitem__method defined. So perhaps you can test for their presense. (These methods may be broken, but OK...)
You misunderstood. The problem was not how to test if something is iterable but how to test if that iterable is a builtin. But, as I said in my previous post, probably this isn't the actual problem anyway. What I want is to deal with strings, lists, tuples and dicts. Period. So, I could be more explicit (and I already tested that the error from comment:44 can be fixed in that way).
comment:51 Changed 3 years ago by
In Python-2, one implemented c.__nonzero__() and then bool(c) would work. It seems that in Python-3, something else needs to be done. Is it perhaps the case that I need to implement __bool__?
What should I do to make stuff work both in py-2 and py-3? Implement both __bool__ and __nonzero__? Or would it be enough to replace __nonzero__ with __bool__?
comment:52 Changed 3 years ago by
comment:53 follow-up: 54 Changed 3 years ago by
There is a problem with the logging module: In Python-2, I get
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: P = CohomologyRing(9,2,options='debug')
We compute this cohomology ring from scratch
Group data are rooted at '/home/king/.sage/temp/klap/19448/dir_0MUJxK/'
Computing basic setup for Small Group number 1 of order 3
Computing basic setup for Small Group number 2 of order 9
Resolution of GF(3)[9gp2]:
> export action matrices
H^*(SmallGroup(9,2); GF(3)):
Initialising maximal p-elementary abelian subgroups
whereas in Python-3 I get
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: P = CohomologyRing(9,2,options='debug') We compute this cohomology ring from scratch Group data are rooted at '/home/king/.sage/temp/klap/18649/dir_9bz1_9bh/' Computing basic setup for Small Group number 1 of order 3 Computing basic setup for Small Group number 2 of order 9 > export action matrices Initialising maximal p-elementary abelian subgroups
So, the indentation is gone, and (whats worse) the logging doesn't indicate what object the log comes from (which I found very useful for debugging, as in the computation of a single cohomology ring often many other rings are involved).
Can someone please give me a pointer to a page explaining the differences between py-2 and py-3 in the logging module?
comment:54 Changed 3 years ago by
Replying to SimonKing:
Can someone please give me a pointer to a page explaining the differences between py-2 and py-3 in the logging module?
By looking at the code of the logging module in Python-2 and Python-3, I see where the problem comes from. In the following, self denotes an instance of pGroupCohomology.auxiliary.CohoFormatter, which sub-classes loggin.Formatter.
In Python-2, logging.Formatter.format(self, record) uses self._fmt to format the output. In Python-3, self._fmt is not used. Instead, self._style._fmt is consulted, where self._style is one of the styles in logging._STYLES.
The point is: So far, my custom self.format modifies self._fmt and then calls the default .format(). However, in Python-3, when calling the default .format(), the attribute self._fmt plays no role, but this is where I currently put the format into.
Simple way out: Do not call logging.Formatter.format in CohoFormatter.format.
But I wonder: Is there a recommended way to make the logging format change from call to call?
By this, I mean: Make the output of CohoFormatter.format(self, record) depend on record.args[0], in the sense that the output makes visible whether record.args[0] is the same object as in the previous call to CohoFormatter.format.
Or, more radically: What kind of logging approach would you recommend, what do you find most easy to comprehend/parse as a human (I am not interested in automatic parsing of the logging messages)?
I somehow like the current approach, which is: Assume that certain objects (say, ZZ and RR) are involved in a computation, and each object may issue a logging message. Then, stream of messages is arranged in blocks of messages coming from a single object, each block being prepended by an identifier of the object. I.e., if ZZ gives two messages, then RR gives three messages and then again ZZ gives a message, the output would be
Integer Ring:
message 1
message 2
Rational Field:
message 3
message 4
message 5
Integer Ring:
message 6
- Do you like this thematic arrangement of messages (which is provided by
coho_loggerin Python 2, but fails in Python-3)? - If yes: Can you recommend a clean and easy way to achieve it both in Python 2 and Python 3? Ideally, one should still be able to include time information in the logging, which currently is not supported.
- If no: What logging format would you recommend?
comment:55 follow-up: 56 Changed 3 years ago by
Instead of setting self._fmt, can you instead run self.basicConfig(format=FORMAT)? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#user-defined-objects), so maybe you can make use of that somehow.
comment:56 Changed 3 years ago by
Replying to jhpalmieri:
Instead of setting
self._fmt, can you instead runself.basicConfig(format=FORMAT)? (https://docs.python.org/3/library/logging.html#logging.basicConfig) There are also ways of having several formatters (https://docs.python.org/3.7/library/logging.config.html#user-defined-objects), so maybe you can make use of that somehow.
From these links, I get the impression that each call to the logger results in the creation of a new formatter. Is that really the case?
comment:57 follow-up: 58 Changed 3 years ago by
I don't know, I only looked at the links, too. It's worth trying.
(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)
comment:58 Changed 3 years ago by
Replying to jhpalmieri:
I don't know, I only looked at the links, too. It's worth trying.
(My impression from the links was that you could create several formatters and then switch from one to another on the fly.)
Sure. But I am unsure how a change of formatter could solve the problem without impact on performance.
If logging is in "warn" mode, then there are of course still silent calls to coho_logger.debug. I want that (1) these calls are cheap (i.e., they shouldn't involve creation of Python objects) and (2) the message blocks should not take into account silent messages, i.e., a block shouldn't be closed if a silent message comes from a different object.
Of course, it should be that what I do with the formatter should rather be done with a handler, or maybe a filter. So, I will try to understand what exactly what handlers and filters do.
And, as in point 3. of comment:54: Would you recommend a different formatting of log messages?
comment:59 Changed 3 years ago by
Currently I am reading logging HOWTO and logging Cookbook, and at the moment it seems to me that I want to use logging.LoggingAdapter, because it allows to add contextual information (which is what I'd like to use, unless people tell me a nicer way log format).
comment:60 Changed 3 years ago by
I think I now know what to do: The customisation should be done in a Formatter, i.e., the existing solution was a good approach. Only the format() method should be different: Before calling super(CohoFormatter, self).format(record), I should add some attribute to the record that is then used in the formatter's format string.
comment:61 Changed 3 years ago by
I think I solved the problem. I can now optionally add time information to the log, such as here:
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: CohomologyRing.global_options('time')
sage: H = CohomologyRing(720,763,prime=3)
2019-09-18 20:54:20,277: Try to compute a Sylow 3-subgroup
2019-09-18 20:54:20,283: Try to find the SmallGroups address of the Sylow subgroup
2019-09-18 20:54:20,288: We compute this cohomology ring from scratch
2019-09-18 20:54:20,289: Group data are rooted at '/home/king/.sage/temp/klap/962/dir_tnhvqeq7/'
2019-09-18 20:54:20,293: Computing basic setup for Small Group number 1 of order 3
2019-09-18 20:54:20,405: Computing basic setup for Small Group number 2 of order 9
Resolution of GF(3)[9gp2]:
2019-09-18 20:54:20,466: > export action matrices
H^*(SmallGroup(9,2); GF(3)):
2019-09-18 20:54:20,517: Initialising maximal p-elementary abelian subgroups
2019-09-18 20:54:20,555: Computing intermediate subgroup
2019-09-18 20:54:20,567: Try to find the SmallGroups address of the intermediate subgroup
2019-09-18 20:54:20,603: Computing group order
2019-09-18 20:54:20,604: The group is of order 72
2019-09-18 20:54:20,609: Computing intermediate subgroup
Resolution of GF(3)[SmallGroup(9,2)]:
2019-09-18 20:54:20,625: Computing next term
2019-09-18 20:54:20,644: > rk P_02 = 3
H^*(SmallGroup(9,2); GF(3)):
2019-09-18 20:54:20,647: We have to choose 2 new generators in degree 1
2019-09-18 20:54:20,648: > There are 2 nilpotent generators in degree 1
...
However, what I'd find even nicer if the sum of the cputimes of Sage and Singular since start of logging was shown, not just the walltime, as above.
Note that the indentation has changed, which means that I will have to change various doc tests, but that will be doable.
comment:62 Changed 3 years ago by
Sigh.
After solving the log problem, I find that meanwhile the code ONLY works in Python-3, not Python-2.
First problem:
class FOO(object):
def __bool__(self):
return something
__nonzero__ = __bool__
works in Py3, but not in Py2, although it was recommended in some stackoverflow discussion. In Py2 it results in NameError: name '__bool__' is not defined. How to fix it without code duplication?
Second problem:
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.py in <module>()
47 import re,os
48
---> 49 import urllib.request, urllib.error
50 import tarfile
51 import logging
ImportError: No module named request
How to deal with urllib in a py2/py3-compatible way?
comment:63 follow-up: 64 Changed 3 years ago by
For the urllib issue: http://python3porting.com/noconv.html#import-errors suggests code like
try:
from urllib.request import ...
except ImportError:
from urllib import ...
I can't reproduce the first problem. With Python 2, if I do
sage: class FOO(object):
....: def __bool__(self):
....: return something
....: __nonzero__ = __bool__
....:
sage: a = FOO()
sage: a.__nonzero__()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-74dab0bd3f82> in <module>()
----> 1 a.__nonzero__()
<ipython-input-1-ec93abb3d539> in __bool__(self)
1 class FOO(object):
2 def __bool__(self):
----> 3 return something
4 __nonzero__ = __bool__
5
NameError: global name 'something' is not defined
so a.__nonzero__() is certainly calling __bool__. If I change the return line to something sensible, it works.
comment:64 Changed 3 years ago by
Replying to jhpalmieri:
For the
urllibissue: http://python3porting.com/noconv.html#import-errors suggests code liketry: from urllib.request import ... except ImportError: from urllib import ...
OK, thank you.
I can't reproduce the first problem. With Python 2, if I do
The "something" was of course figuratively. Anyway, the problem seems to be that I am using a cdef class here:. In Sage-with-py2:
sage: cython('''
....: class Foo:
....: def __bool__(self):
....: return True
....: __nonzero__ = __bool__
....: ''')
sage: cython('''
....: cdef class Foo:
....: def __bool__(self):
....: return True
....: __nonzero__ = __bool__
....: ''')
ERROR:root:An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line string', (1, 0))
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-2-91e48d391924> in <module>()
4 return True
5 __nonzero__ = __bool__
----> 6 ''')
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/lazy_import.pyx in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3697)()
352 True
353 """
--> 354 return self.get_object()(*args, **kwds)
355
356 def __repr__(self):
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_compile(code, **kwds)
649 with open(tmpfile,'w') as f:
650 f.write(code)
--> 651 return cython_import_all(tmpfile, get_globals(), **kwds)
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_import_all(filename, globals, **kwds)
539 code
540 """
--> 541 m = cython_import(filename, **kwds)
542 for k, x in iteritems(m.__dict__):
543 if k[0] != '_':
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/cython.pyc in cython_import(filename, **kwds)
519 try:
520 sys.path.append(build_dir)
--> 521 return builtins.__import__(name)
522 finally:
523 sys.path = oldpath
/home/king/.sage/temp/klap/23274/spyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx/_home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0.pyx in init _home_king__sage_temp_klap_23274_tmp_FgSymA_pyx_0()
3 def __bool__(self):
4 return True
----> 5 __nonzero__ = __bool__
NameError: name '__bool__' is not defined
whereas in Sage-with-py3 I get
sage: cython('''
....: class Foo:
....: def __bool__(self):
....: return True
....: __nonzero__ = __bool__
....: ''')
sage: cython('''
....: cdef class Foo:
....: def __bool__(self):
....: return True
....: __nonzero__ = __bool__
....: ''')
So, what to do?
comment:65 follow-up: 66 Changed 3 years ago by
I don't know Cython, but replacing __nonzero__ = __bool__ with
def __nonzero__(self):
return self.__bool__()
seems to work.
comment:66 Changed 3 years ago by
Replying to jhpalmieri:
I don't know Cython, but replacing
__nonzero__ = __bool__withdef __nonzero__(self): return self.__bool__()seems to work.
Sure. that's what I basically meant with "code duplication". It is an additional call, thus, if the boolean value of a cochain is tested very often then one will see a slow-down. But I don't know if it actually is tested very often.
Anyway, with this code cuplication, things seem to work. Hooray!
At the moment, it seems that most of the remaining work will be to change the expected output for those tests that use logging.
comment:67 follow-up: 68 Changed 3 years ago by
My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does __nonzero__ do anything with Python 3? If so, maybe you could do something like
from six import PY2
if PY2:
def __nonzero__(self):
...
to avoid the extra call when using Python 3.
comment:68 follow-up: 81 Changed 3 years ago by
Replying to jhpalmieri:
My hope is that we can drop Python 2 support pretty soon, or at least make sure that Python 3 works well, even if it as the expense of performance with Python 2. Does
__nonzero__do anything with Python 3? If so, maybe you could do something likefrom six import PY2 if PY2: def __nonzero__(self): ...to avoid the extra call when using Python 3.
I don't think that __nonzero__ does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current not-yet-published code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for __nonzero__ and for __bool__, but only one of them would be compiled.
comment:69 Changed 3 years ago by
One detail: According to unit_test_64, there is no significant speed difference between the py2 and py3 versions. In any case, the cputime obtained as sum of the cputime() and Singular's timer is only half of the walltime. Do I recall correctly that the time used to save data on disk is not measured by cputime()? Is there an easy way to determine that time, too?
comment:70 Changed 3 years ago by
I don't think it makes much sense measuring time to save to disk, as it varies wildly between different OS's, different hardware, different configuration of buffering, etc etc.
Often saving is completely asyncronous from the computation due to buffering and separate threads run to do buffer flushing, etc.
comment:71 Changed 3 years ago by
Now I am totally baffled. Is it not the case that comparison is supposed to be implemented by __richcmp__? Apparently __richcmp__ isn't called anymore.
The following corresponds to the code in pGroupCohomology.dickson, that did work in the past:
sage: cython("""
....: from sage.structure.richcmp import richcmp, op_LT, op_NE, op_GT
....: from sage.all import GF
....: class DICKSON:
....: def __init__(self,p):
....: self.K = GF(p)
....: def __richcmp__(self, other, op):
....: print("Comparison called")
....: return richcmp(self.K, other.K, op)
....: """)
Now, rich comparison fails both in py2 and py3:
sage: D = DICKSON(5) sage: F = loads(dumps(D)) sage: D == F False sage: F.K == D.K True
What has changed, how to fix it?
comment:72 follow-up: 73 Changed 3 years ago by
Aha! I see in sage.misc.unknown that a decorator @richcmp_method is used. So, I guess I need to use it, too.
comment:73 Changed 3 years ago by
Replying to SimonKing:
Aha! I see in sage.misc.unknown that a decorator
@richcmp_methodis used. So, I guess I need to use it, too.
Hooray:
sage: cython("""
....: from sage.structure.richcmp import richcmp, richcmp_method
....: from sage.all import GF
....: @richcmp_method
....: class DICKSON(object):
....: def __init__(self,p):
....: self.K = GF(p)
....: def __richcmp__(self, other, op):
....: print("Comparison called")
....: return richcmp(self.K, other.K, op)
....: """)
sage: D = DICKSON(5)
sage: F = loads(dumps(D))
sage: D == F
Comparison called
True
comment:74 follow-ups: 75 76 Changed 3 years ago by
At the moment, tests in dickson, auxiliaries, resolution and factory pass both in Py2 and Py3. In isomorphism_test, there are currently two failing tests, and it seems to me that they reveal a Py3-bug in cartesian_product_iterator:
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 353, in pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms
Failed example:
T.explore_isomorphisms()
Exception raised:
Traceback (most recent call last):
File "sage/misc/cachefunc.pyx", line 1944, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10139)
return cache[k]
KeyError: (((3, 5, 4, 6, 7, 1, 2), 0, True), ())
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/mrange.py", line 139, in _xmrange_iter
curr_elt = [next(i) for i in curr_iters[:-1]] + [None]
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/misc/mrange.py", line 139, in <listcomp>
curr_elt = [next(i) for i in curr_iters[:-1]] + [None]
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
exec(compiled, globs)
File "<doctest pGroupCohomology.isomorphism_test.IsomorphismTest.explore_isomorphisms[10]>", line 1, in <module>
T.explore_isomorphisms()
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/isomorphism_test.py", line 402, in explore_isomorphisms
Cands = self.candidates_of_gens(tuple([p[4] for p in Sizes]),len(self.rigid_generators))
File "sage/misc/cachefunc.pyx", line 1949, in sage.misc.cachefunc.CachedMethodCaller.__call__ (build/cythonized/sage/misc/cachefunc.c:10273)
w = self._instance_call(*args, **kwds)
File "sage/misc/cachefunc.pyx", line 1825, in sage.misc.cachefunc.CachedMethodCaller._instance_call (build/cythonized/sage/misc/cachefunc.c:9758)
return self.f(self._instance, *args, **kwds)
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/pGroupCohomology/isomorphism_test.py", line 644, in candidates_of_gens
for Ims1,Ims2 in cartesian_product_iterator([FirstHalf, SecondHalf]):
RuntimeError: generator raised StopIteration
I will investigate which values of FirstHalf and SecondHalf trigger it.
comment:75 Changed 3 years ago by
comment:76 Changed 3 years ago by
Replying to SimonKing:
At the moment, tests in
dickson,auxiliaries,resolutionandfactorypass both in Py2 and Py3.
When working around the cartesian_product_iterator bug, the tests in isomorphism_test pass both in Py2 and Py3. Yay!
comment:77 Changed 3 years ago by
Two errors in Py3 concern __div__, namely
Running doctests with ID 2019-09-21-07-26-57-20b48344.
Using --optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage
Doctesting 1 file.
sage -t --long --warn-long 47.4 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx", line 3131, in pGroupCohomology.cochain.MODCOCH.__div__
Failed example:
print(H.1/3) #indirect doctest
Expected:
(a_6_0)/(3): 6-Cocycle in H^*(SmallGroup(400,206); GF(5))
defined by
2*c_2_1*c_2_2*a_1_0*a_1_1
Got:
(a_6_0)*(2): 6-Cocycle in H^*(SmallGroup(400,206); GF(5))
defined by
2*c_2_1*c_2_2*a_1_0*a_1_1
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx", line 3139, in pGroupCohomology.cochain.MODCOCH.__div__
Failed example:
H.2/H.1
Expected:
Traceback (most recent call last):
...
TypeError: Can not divide by <class 'pGroupCohomology.cochain.MODCOCH'>
Got:
<BLANKLINE>
Traceback (most recent call last):
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/king/Sage/git/py3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1123, in compile_and_execute
exec(compiled, globs)
File "<doctest pGroupCohomology.cochain.MODCOCH.__div__[6]>", line 1, in <module>
H.gen(2)/H.gen(1)
File "sage/structure/element.pyx", line 1718, in sage.structure.element.Element.__truediv__ (build/cythonized/sage/structure/element.c:12780)
return (<Element>left)._div_(right)
File "sage/structure/element.pyx", line 2654, in sage.structure.element.RingElement._div_ (build/cythonized/sage/structure/element.c:18184)
frac = self._parent.fraction_field()
TypeError: 'NoneType' object is not callable
**********************************************************************
1 item had failures:
2 of 8 in pGroupCohomology.cochain.MODCOCH.__div__
[1490 tests, 2 failures, 465.54 s]
----------------------------------------------------------------------
sage -t --long --warn-long 47.4 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx # 2 doctests failed
----------------------------------------------------------------------
Total time for all tests: 470.6 seconds
cpu time: 249.3 seconds
cumulative wall time: 465.5 seconds
The first error seems to indicate that H.1/3 is automatically translated to H.1*2, which does make sense modulo 5, but which I did not implement like this. And the second error is nothing that I implemented.
In fact I overrode __div__ for COCH and MODCOCH. Is it perhaps the case that overriding __div__ won't work in Py3?
comment:79 follow-up: 80 Changed 3 years ago by
Replying to chapoton:
In py3, there is
__truediv__and__floordiv__
I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement __truediv__, __floordiv__ and __div__?
On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.
So, given a RingElement, what methods do I need to implement for providing division by an element of the base field, and what do I need to implement for making clear that division of two ring elements doesn't work?
I guess I should implement a fraction_field method for cohomology rings. Of course, since cohomology rings can have zero divisors, a fraction field will not always exist. So, should in that situation fraction_field return NotImplemented, or raise a ValueError, or what else?
Do I need to provide a single-underscore _div_ method, in addition to the fraction_field?
How to implement division by a base field element (I guess _div_ is for division of two ring elements)?
comment:80 Changed 3 years ago by
Replying to SimonKing:
Replying to chapoton:
In py3, there is
__truediv__and__floordiv__I see. Do I understand correctly that in order to make my code Py2 and Py3 compatible, I should implement
__truediv__,__floordiv__and__div__?On the other hand, I reckon that Sage has default implementations of the double underscore methods, that I should rather not override.
I think I'll proceed as follows:
- Implement a
fraction_fieldmethod for cohomology rings raising a TypeError (note thatZZ.quo(6).fraction_field()raises a TypeError, too) - Do NOT implement
_div_-- the attempt to divide two elements of a cohomology ring will result in the TypeError raised by thefraction_field()method - Division of a cohomology element by an element of the base field will automatically work, by multiplying the inverse of the field element with the cohomology element, using
_lmul_/_rmul_. I should add an example/test in the documentation of the_lmul_/_rmul_methods
comment:81 Changed 3 years ago by
Replying to SimonKing:
I don't think that
__nonzero__does anything in Python 3. But now as you mention it: It is possible (and is actually used in some places in the cythonised parts of the Sage library and also in the current not-yet-published code of the cohomology package) to make the code depend on the language level. I.e., I could have definitions for__nonzero__and for__bool__, but only one of them would be compiled.
This did work, to some extent. But not really reliably. Therefore, to be on the safe side, I will implement both methods in both python versions, since the code duplication is only small.
comment:82 Changed 3 years ago by
I am currently dealing with different but mathematically equivalent behaviour: Apparently, previously Singular did not always do tail reduction, i.e., some rings in the database do not have a fully reduced representation of the relation ideal. For example, here are relations for the cohomology of SmallGroup(64,14):
sage: H.rels() # not tail reduced, result obtained with py2 ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1^2', 'a_1_0*a_3_3+a_2_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
versus
sage: H.rels() # tail reduced, obtained with Py3 ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_0*a_3_3+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1*a_1_1^2', 'a_2_1*a_3_3', 'a_3_3^2']
As you can see, the relation a_2_1^2+a_2_1*a_1_1^2 could be used to reduce the tail of a_1_1*a_3_3+a_2_1^2. Also, the sorting of the relations is different.
The question is how to deal with it. I guess I could proceed as follows:
sage: if (2, 8) < sys.version_info: ....: expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', ....: 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_0*a_3_3+a_2_1*a_1_1^2', ....: 'a_1_1*a_3_3+a_2_1*a_1_1^2', 'a_2_1*a_3_3', 'a_3_3^2'] ....: else: ....: expected_rels = ['a_1_0^2', 'a_1_0*a_1_1', 'a_1_1^3', 'a_2_1*a_1_0', ....: 'a_2_1^2+a_2_1*a_1_1^2', 'a_1_1*a_3_3+a_2_1^2', ....: 'a_1_0*a_3_3+a_2_1^2', 'a_2_1*a_3_3', 'a_3_3^2'] ....: sage: H.rels() == expected_rels True
comment:83 Changed 3 years ago by
Sigh. I am suffering from "interesting" debugging sessions. At the moment, it seems to me that some errors in the py2-version only occur, when I first install p_group_cohomology in the py2-install of Sage, then install p_group_cohomology in the py3-install of Sage, and only then do tests in the py2-install.
comment:84 follow-up: 85 Changed 3 years ago by
I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of ~/.sage/.
comment:85 follow-up: 86 Changed 3 years ago by
Replying to dimpase:
I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of
~/.sage/.
You mean by testing the p_group_cohomology package, or something else?
Indeed I had the same impression concerning ~/.sage/. Fortunately most of the trouble could be solved by putting str() around some data (because I got unicode where a string was expected - so, perhaps a rogue py3-pickle got misinterpreted in py2).
comment:86 Changed 3 years ago by
Replying to SimonKing:
Replying to dimpase:
I had a bad experience with doing py2/3 things on the same account, the problems was apparently in the content of
~/.sage/.You mean by testing the p_group_cohomology package, or something else?
Indeed I had the same impression concerning
~/.sage/. Fortunately most of the trouble could be solved by puttingstr()around some data (because I got unicode where a string was expected - so, perhaps a rogue py3-pickle got misinterpreted in py2).
I was getting different output from a test, IIRC it was releated to unicode, and the problem went away after I removed ~/.sage/. I thought it might have been related to ipython, but who knowns.
comment:87 Changed 3 years ago by
I tried to solve it by doing an explicit conversion to str whenever there is data that is expected of this type. And instead of if isinstance(bla, str), I now do if isinstance(bla, (str, unicode)), where I define unicode=str in py3.
So far, this seems to work. And after all, most users will have only one Sage installation anyway.
comment:88 Changed 3 years ago by
Replying to SimonKing:
Replying to SimonKing:
If the .sobj file's location is not stored as an attribute in Python 3 then I have a problem.
Hooray:
# in Sage-with-python-2 sage: P = CombinatorialFreeModule(ZZ,'x') sage: save(P,'../../pickle')# BOTH with python-2 and python-3 sage: B = load('../../pickle.sobj') sage: B._default_filename '/home/king/Sage/pickle.sobj'So, the old trick should still work.
I wonder if I shouldn't improve the trick.
Currently, if the initialisation of an unpickled object can not be finished, it is the __getattr__ method that would trigger the completion of initialisation based on the value of self._default_filename that is set by Sage's loads() after unpickling the object and before returning it.
But what if I turn _default_filename into a property? I could make it so that its setter triggers the completion of initialisation, to the effect that the object returned by loads() will be valid. The trickery would be in the property setter, not in __getattr__.
comment:89 Changed 3 years ago by
I am puzzled about the appearance of unicode. For example, consider these two lines in my code:
root = str(kwds.get('root', COHO.workspace)) coho_logger.debug("Group data are rooted at %r", None, root)
The logging appears in the following session (in Sage-with-py2, where I first installed the package in Sage-with-py3 and then in Sage-with-py2):
sage: from pGroupCohomology import CohomologyRing
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: from pGroupCohomology.cohomology import COHO
sage: _ = COHO()
<module>:
Group data are rooted at u'/home/king/.sage/temp/klap/6618/dir_T5Im2o/'
So, apparently the variable root in the code snipped above is a unicode, although I explicitly converted it to a str. Moreover:
sage: COHO.workspace '/home/king/.sage/temp/klap/6618/dir_T5Im2o/' sage: type(COHO.workspace) <type 'str'>
In other words: There is an attribute COHO.workspace of type str, and when I convert it to a str then a unicode results. WTF???
comment:90 follow-up: 91 Changed 3 years ago by
I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.
It might be that iPython installs some kind of display hook that sets str to convert to unicode. There is code in src/sage/repl/rich_output that might be doing this.
That is to say that unicode you see printed might be happening due to on the fly conversion, set up by DisplayManager.
comment:91 Changed 3 years ago by
Replying to dimpase:
I don't see a full picture here, as it's unclear which sessions are in py2 and which are in py3.
Is there unicode in py3?
The common topic is: In some cases I get a TypeError saying that a str was expected, but got unicode. And a log shows that a variable is of type unicode after an EXPLICIT(!) cast to str.
In all mentioned cases, I proceeded as follows:
- Install the package in Sage-with-py2
- Install the package in Sage-with-py3
- Test in Sage-with-py2
Sage-with-py2 and Sage-with-py3 are of course in different folders. What they share: The git repository (but different git worktrees), and DOT_SAGE.
But it is all very flaky. I just inserted a couple of print statements in my code to investigate more deeply, did the above double-install procedure --- and suddenly the type was str, not unicode.
It might be that iPython installs some kind of display hook that sets
strto convert to unicode. There is code insrc/sage/repl/rich_outputthat might be doing this.
It is not only the output but the type inside of the code. In some cases, I get a TypeError.
comment:92 Changed 3 years ago by
Do you see these errors while running doctests? I gather there is --nodotsage option you can use while running tests, so you may try to see whether you can reproduce this without DOT_SAGE.
Another thing to try is to wipe out DOT_SAGE each time you switch from pyX to py(5-X), and see whether this make the error to go away.
comment:93 Changed 3 years ago by
another way to work around this might be just to convert all your strings to unicode.
comment:94 Changed 3 years ago by
I am afraid that it is currently not possible for anybody else to reproduce the following, because I didn't publish my code yet. But anyway, here is what I did:
- Insert type information, so that the mentioned code snippet now is
root = str(kwds.get('root', COHO.workspace)) coho_logger.debug("Group data are rooted at %r. Type: %r", None, root, type(root))
- Install it in py3
- Install it in py2
- In Py3, run the following:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: H = CohomologyRing(64,14, from_scratch=True) sage: H.make()
- In Py2, run the following:
sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() <module>: Group data are rooted at u'/home/king/.sage/temp/klap/14917/dir_20mizE/'. Type: <type 'unicode'>
So, the type REALLY is unicode, even though I did an EXPLICIT conversion to str.
Then, I continued:
- Move the old
~/.sageand create a new one. - Start Sage-with-py2, and repeat the test:
Setting permissions of DOT_SAGE directory so only you can read and write it. sage: from pGroupCohomology import CohomologyRing sage: from pGroupCohomology.cohomology import COHO sage: CohomologyRing.doctest_setup() sage: CohomologyRing.global_options('debug') sage: _ = COHO() <module>: Group data are rooted at u'/home/king/.sage/temp/klap/15112/dir_fYCkwo/'. Type: <type 'unicode'> sage: type(COHO.workspace) <type 'str'>
So, removing DOT_SAGE didn't help. For comparison, do the same in Py3:
sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
<module>:
Group data are rooted at '/home/king/.sage/temp/klap/15493/dir_81mwix3u/'. Type: <class 'str'>
which of course is no surprise, as python-3 has no unicode.
Note that the line _ = COHO() does not involve taking an old pickle from a repository. It merely creates a new instance of COHO and initialises it only to the point where the current workspace is assigned to the attribute .root.
Summary: The class attribute COHO.workspace is of type str. The dictionary kwds is empty. str(kwds.get('root', COHO.workspace))) returns the unicode version of `COHO.workspace. I.e., converting a str to a str results in a unicode.
comment:95 Changed 3 years ago by
Just to document that we are talking about a Heisenbug: I wanted to test whether it is perhaps the case that str==unicode. Therefore, I changed the code snippet to
root = str(kwds.get('root', COHO.workspace))
coho_logger.debug("Group data are rooted at %r. Type: %r, %r, %r", None, root, type(root), type(COHO.workspace), str)
, installed it in py2 (only), and rerun the test in py2:
sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
<module>:
Group data are rooted at '/home/king/.sage/temp/klap/16014/dir_M5saF0/'. Type: <type 'str'>, <type 'str'>, <type 'str'>
Hence, no bug. "Hooray".
Start a sage-with-py3-session and do
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing.doctest_setup() sage: H = CohomologyRing(64,14, from_scratch=True) sage: H.make()
The hope was of course that it triggers something. But the test in py2 does not have a unicode problem. "Hooray".
But I am sure that the problem will pop up sooner or later.
comment:96 Changed 3 years ago by
To make sure that this bug doesn't pop up again, I inserted a test whether the python version is less than (2,8) and root is of type unicode. In this case, I print some information and then raise a runtime error. I guess it's the best that I can do for now.
After recompiling and rerunning the above tests, they went fine again. But sooner or later it will return, and then the diagnostic tests I inserted will perhaps tell more clearly what is happening.
comment:97 Changed 3 years ago by
Or another idea: Since it is Cython code, I could do cdef str root = ....
comment:98 Changed 3 years ago by
Hooray!
The inserted test already showed something:
sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
Critical bug in python.
kwds.get('root', COHO.workspace) is <type 'str'>
COHO.workspace is <type 'str'>
str=<type 'unicode'>, unicode=<type 'unicode'>
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-5-a303ce041f27> in <module>()
----> 1 _ = COHO()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()
RuntimeError: An explicit cast to <str> resulted in a unicode object
Aha! For some silly reason, we have str == unicode!!
EDIT: The above was printed like this:
if (2, 8) > sys.version_info and isinstance(root, unicode): print("Critical bug in python.") print("kwds.get('root', COHO.workspace) is %r"%type(kwds.get('root', COHO.workspace))) print("COHO.workspace is %r"%type(COHO.workspace)) print("str=%r, unicode=%r"%(str,unicode)) raise RuntimeError("An explicit cast to <str> resulted in a unicode object")
comment:99 Changed 3 years ago by
How to cope with it? Shall I test upon importing of the module whether str == unicode in Python-2, and raise an error whose message tells to recompile?
comment:100 follow-up: 101 Changed 3 years ago by
So, would this still pop up if right before running these tests you removed ~/.sage/ ?
comment:101 Changed 3 years ago by
Replying to dimpase:
So, would this still pop up if right before running these tests you removed
~/.sage/?
See comment:94. There, it didn't help. Now, it doesn't either:
king@klap:~/Sage/git$ mv ~/.sage/ ~/.sageBAK
king@klap:~/Sage/git$ mkdir ~/.sage
king@klap:~/Sage/git$ sage/sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9.beta8, Release Date: 2019-08-25 │
│ Using Python 2.7.15. Type "help()" for help. │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Setting permissions of DOT_SAGE directory so only you can read and write it.
sage: from pGroupCohomology import CohomologyRing
sage: from pGroupCohomology.cohomology import COHO
sage: CohomologyRing.doctest_setup()
sage: CohomologyRing.global_options('debug')
sage: _ = COHO()
Critical bug in python.
kwds.get('root', COHO.workspace) is <type 'str'>
COHO.workspace is <type 'str'>
str=<type 'unicode'>, unicode=<type 'unicode'>
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-5-a303ce041f27> in <module>()
----> 1 _ = COHO()
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__init__()
RuntimeError: An explicit cast to <str> resulted in a unicode object
comment:102 Changed 3 years ago by
I wonder if the problem occurs because I am not installing the development versions of my package from an actual "package" (using sage -i p_group_cohomology), but directly from the sources. I.e., in the src folder of my development tree, I do
sage -python setup.py build sage -pip install . --upgrade
I know that sage -python setup.py build does create .c files in the src folder, but since the changed module is recompiled both in py2 and py3, I thought that it doesn't matter. And for efficiency, I want that only modules are recompiled that have changed.
So, I would like to test if the problem persists when separating the py2- and py3 build folders. When doing sage -python setup.py build, is it somehow possible to prescribe in what folder the .c and .pyc files are created? Then, I could choose separate folders for py2 and py3.
comment:103 Changed 3 years ago by
I am now putting cythonize(..., build_dir=os.path.join("build","c_files-{}.{}".format(version_info.major, version_info.minor))) into my setup.py. In that way, the c-files will be created in separate directories for both Sage versions.
Let's see if this is enough to solve the problem!
comment:104 Changed 3 years ago by
PS: Even if it solves the problem, I will test upon import whether str is unicode in py2, and if it is, raise a RuntimeError advising to recompile. I hope this error will never pop up...
comment:105 Changed 3 years ago by
After changing setup.py so that the .c files will be in different folders for different python versions, tests in cohomology.pyx pass both with py2 and py3 and then py2 again. So, I can now turn to modular_cohomology.pyx and finally init.py (which contains a long doc string, namely the "general overview" of the package).
comment:106 Changed 3 years ago by
Hooray! When I ran sage -tp 2 --force-lib --long --warn-long 60.1, all tests passed both in python 2 and python 3. I don't know how significant the run time differences are. In any case, here are the details:
Python-3
========
Running doctests with ID 2019-09-25-00-50-30-78efa273.
Using --optional=build,dochtml,gap_packages,libsemigroups,meataxe,memlimit,mpir,python2,sage
Sorting sources by runtime so that slower doctests are run first....
Doctesting 13 files using 2 threads.
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pxd
[0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
[1481 tests, 350.33 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cohomology.pyx
[1445 tests, 363.25 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
Warning, slow doctest:
T.explore_isomorphisms() # long time
Test ran for 122.79 s
[213 tests, 221.05 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/modular_cohomology.pyx
[459 tests, 274.13 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/factory.py
[229 tests, 126.99 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/__init__.py
[343 tests, 96.76 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/dickson.pyx
[28 tests, 1.53 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/auxiliaries.py
[39 tests, 0.79 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/barcode.py
[143 tests, 10.31 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pxd
[0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution_bindings.pxd
[0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pyx
[882 tests, 22.95 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 765.7 seconds
cpu time: 939.6 seconds
cumulative wall time: 1468.1 seconds
Python-2
========
Running doctests with ID 2019-09-25-01-04-47-fa1de7f4.
Git branch: matrix_implementation_keyword
Using --optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,p_group_cohomology,python2,sage
Sorting sources by runtime so that slower doctests are run first....
Doctesting 13 files using 2 threads.
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pyx
[1481 tests, 437.44 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cochain.pxd
[0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/cohomology.pyx
[1445 tests, 465.15 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py
**********************************************************************
File "/home/king/Projekte/coho/coho-devel/src/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest
Warning, slow doctest:
T.explore_isomorphisms() # long time
Test ran for 116.79 s
[213 tests, 229.85 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/modular_cohomology.pyx
[459 tests, 312.40 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/factory.py
[229 tests, 124.88 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/__init__.py
[343 tests, 96.86 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pxd
[0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution.pyx
[882 tests, 21.38 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/dickson.pyx
[28 tests, 1.63 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/auxiliaries.py
[39 tests, 0.78 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/resolution_bindings.pxd
[0 tests, 0.00 s]
sage -t --long --warn-long 60.1 /home/king/Projekte/coho/coho-devel/src/pGroupCohomology/barcode.py
[143 tests, 14.90 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 873.5 seconds
cpu time: 996.8 seconds
cumulative wall time: 1705.3 seconds
Hence, tomorrow (i.e., in central European time, today in a few hours :-)) I will create a new package version and post it here for review.
comment:107 Changed 3 years ago by
Something more to do: I think I should add a test showing what happens when moving the directory associated with a cohomology ring. The package can cope with moved data, based on some trickery in the attribute _default_filename that is defined by Sage's unpickler. I've turned _default_filename into a property (with getter, setter and deleter), and its documentation should contain the new test.
comment:108 Changed 3 years ago by
| Description: | modified (diff) |
|---|
I'm afraid that another test failure popped up. So, I cannot post the new version yet.
comment:109 Changed 3 years ago by
After fixing the failed test (in which I forgot to save something before loading it), I added a new feature to unit_test_64: Previously it was only tested whether the generator degrees and the Poincaré series of a freshly computed cohomology ring is consistent with what is stored in the database; now I test whether the new ring and the stored ring are isomorphic.
Indeed they are, for all groups of order 64. More precisely: In many cases the generators of the relation ideal are not identical. But in all cases, mapping the i-th generator of the new ring to the i-th generator of the old ring results in an isomorphism.
There is one remaining py3 issue, though: When I try to build the documentation, I get the error "No module named sphinx". The documentation does build, when I use py2. What can I do to fix it in py3?
EDIT: Is it not the case that sphinx is installed in Sage for building its documentation, both in py2 and py3? I am not sure if I ever tried to build the documentation of Sage-with-py3, though. Will sphinx be installed when I do a plain make in Sage (i.e., when I build the docs)?
comment:110 follow-up: 111 Changed 3 years ago by
Sphinx should be installed, yes. Look for SAGE_ROOT/local/lib/python3.7/site-packages/sphinx/. If you run Sage, can you import sphinx?
Edit: this is for a Python 3 build of Sage.
comment:111 Changed 3 years ago by
Replying to jhpalmieri:
Sphinx should be installed, yes. Look for
SAGE_ROOT/local/lib/python3.7/site-packages/sphinx/. If you run Sage, can youimport sphinx?
It was a trivial mistake: In the doc's Makefile, I had SPHINXBUILD = python -msphinx, but since python means python-2 even in a py3-install of Sage, I have to use SPHINXBUILD = sage -python -msphinx
comment:112 Changed 3 years ago by
| Branch: | → u/SimonKing/p_group_cohomology_in_py_3 |
|---|
comment:113 Changed 3 years ago by
| Authors: | → Simon King |
|---|---|
| Commit: | → 463806c6cebb3120e20df0f19a216dd529553741 |
| Description: | modified (diff) |
| Status: | new → needs_review |
I checked that the package builds and passes tests in Sage-with-py3. I checked that when running the tests directly on the sources (not during package installation) the tests pass in Sage-with-py2. I recommend that whoever reviews this tries the package both in py2 and in py3.
New commits:
| 2e0d11e | Pass unpickling options to pickle.load, default encoding 'latin1'. Accept both str and bytes in mtx_unpickle
|
| 8c105cd | Make str_to_bytes/bytes_to_str accept both str and bytes input.
|
| c8a0748 | Add tests for #28444
|
| 89ac77a | Fix keyword for py3-only test
|
| 3693024 | Fix doc strings in sage.misc.persist
|
| f0828ee | Add a comment regarding the expected data type for an unpickle helper
|
| ba41ebe | Fix two typos in a comment
|
| 463806c | Upgrade p_group_cohomology to version 3.3
|
comment:114 follow-up: 115 Changed 3 years ago by
Hooray! In Py2, sage -f -c p_group_cohomology results in
... [p_group_cohomology-3.3] Running the test suite for p_group_cohomology-3.3... [p_group_cohomology-3.3] make[2]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1' [p_group_cohomology-3.3] Making check in src [p_group_cohomology-3.3] make[3]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] make mnttest.sh [p_group_cohomology-3.3] make[4]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] test -r ./test.reg || cp ./test.reg . [p_group_cohomology-3.3] make[4]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] make check-TESTS [p_group_cohomology-3.3] make[4]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] make[5]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] PASS: mnttest.sh [p_group_cohomology-3.3] ============================================================================ [p_group_cohomology-3.3] Testsuite summary for modular_resolution 1.1 [p_group_cohomology-3.3] ============================================================================ [p_group_cohomology-3.3] # TOTAL: 1 [p_group_cohomology-3.3] # PASS: 1 [p_group_cohomology-3.3] # SKIP: 0 [p_group_cohomology-3.3] # XFAIL: 0 [p_group_cohomology-3.3] # FAIL: 0 [p_group_cohomology-3.3] # XPASS: 0 [p_group_cohomology-3.3] # ERROR: 0 [p_group_cohomology-3.3] ============================================================================ [p_group_cohomology-3.3] make[5]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] make[4]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] make[3]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1/src' [p_group_cohomology-3.3] make[3]: Entering directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1' [p_group_cohomology-3.3] make[3]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1' [p_group_cohomology-3.3] make[2]: Leaving directory '/home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3/src/modular_resolution-1.1' [p_group_cohomology-3.3] Running doctests with ID 2019-09-26-11-19-11-b01ece2b. [p_group_cohomology-3.3] Git branch: t/28414/p_group_cohomology_in_py_3 [p_group_cohomology-3.3] Using --optional=build,ccache,dochtml,frobby,gap_packages,gdb,gfortran,libsemigroups,meataxe,memlimit,mpir,python2,sage [p_group_cohomology-3.3] Sorting sources by runtime so that slower doctests are run first.... [p_group_cohomology-3.3] Doctesting 13 files using 3 threads. [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/modular_cohomology.pyx [p_group_cohomology-3.3] [459 tests, 456.02 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/cochain.pxd [p_group_cohomology-3.3] [0 tests, 0.00 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/cochain.pyx [p_group_cohomology-3.3] [1481 tests, 599.35 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/cohomology.pyx [p_group_cohomology-3.3] [1465 tests, 606.86 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/isomorphism_test.py [p_group_cohomology-3.3] ********************************************************************** [p_group_cohomology-3.3] File "pGroupCohomology-3.3/pGroupCohomology/isomorphism_test.py", line 67, in pGroupCohomology.isomorphism_test.IsomorphismTest [p_group_cohomology-3.3] Warning, slow doctest: [p_group_cohomology-3.3] T.explore_isomorphisms() # long time [p_group_cohomology-3.3] Test ran for 161.47 s [p_group_cohomology-3.3] [213 tests, 307.72 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/__init__.py [p_group_cohomology-3.3] [343 tests, 148.33 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/factory.py [p_group_cohomology-3.3] ********************************************************************** [p_group_cohomology-3.3] File "pGroupCohomology-3.3/pGroupCohomology/factory.py", line 133, in pGroupCohomology.factory.unit_test_64 [p_group_cohomology-3.3] Warning, slow doctest: [p_group_cohomology-3.3] L, t = unit_test_64(from_scratch=False) # long time [p_group_cohomology-3.3] Test ran for 99.46 s [p_group_cohomology-3.3] [229 tests, 209.64 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/resolution.pxd [p_group_cohomology-3.3] [0 tests, 0.00 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/resolution.pyx [p_group_cohomology-3.3] [882 tests, 26.85 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/dickson.pyx [p_group_cohomology-3.3] [28 tests, 1.41 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/barcode.py [p_group_cohomology-3.3] [143 tests, 21.04 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/resolution_bindings.pxd [p_group_cohomology-3.3] [0 tests, 0.00 s] [p_group_cohomology-3.3] sage -t --long --warn-long 80.0 pGroupCohomology-3.3/pGroupCohomology/auxiliaries.py [p_group_cohomology-3.3] [39 tests, 0.92 s] [p_group_cohomology-3.3] ---------------------------------------------------------------------- [p_group_cohomology-3.3] All tests passed! [p_group_cohomology-3.3] ---------------------------------------------------------------------- [p_group_cohomology-3.3] Total time for all tests: 815.2 seconds [p_group_cohomology-3.3] cpu time: 1261.3 seconds [p_group_cohomology-3.3] cumulative wall time: 2378.1 seconds [p_group_cohomology-3.3] [p_group_cohomology-3.3] real 13m52.051s [p_group_cohomology-3.3] user 22m48.984s [p_group_cohomology-3.3] sys 4m1.098s [p_group_cohomology-3.3] Successfully installed p_group_cohomology-3.3 [p_group_cohomology-3.3] Deleting temporary build directory [p_group_cohomology-3.3] /home/king/Sage/git/sage/local/var/tmp/sage/build/p_group_cohomology-3.3 [p_group_cohomology-3.3] Finished installing p_group_cohomology-3.3.spkg make[1]: Leaving directory '/home/king/Sage/git/sage/build/make' real 27m19.468s user 30m7.398s sys 4m10.930s Sage build/upgrade complete!
In Py3, I had more warnings about slow tests (six instead of two, if I recall correctly), but perhaps my laptop (or its harddisk) was busy with other things.
Now I am running unit_test_64 in Python-2. Note that in Python-3 I got about 20 (or even more) examples where the freshly computed relation ideal was presented by a different set of generators than what was stored in the database. But, as I have mentioned in comment:82, this seems to be Singular not doing all tail reduction in py2.
comment:115 Changed 3 years ago by
Replying to SimonKing:
Now I am running
unit_test_64in Python-2. Note that in Python-3 I got about 20 (or even more) examples where the freshly computed relation ideal was presented by a different set of generators than what was stored in the database.
Just for the record: In the py2 version, the freshly computed results were identical (not only equal up to generator choice) to the stored results.
comment:116 Changed 3 years ago by
| Description: | modified (diff) |
|---|
By the way, at the recent Sage Days in Bonn, I learnt how to host the package repository at github and use Travis CI to some extent. I have posted the link to the ticket description.
Note, however, that testing it on Travis CI with python-3 will be difficult: I was told that the current Sage-with-py3 docker image is rather outdated, and also I don't know how I should write my .travis.yml so that the automatic tests involve both py2 and py3. The answer on sage-devel didn't help me much. Is someone reading this who can tell me how I should modify .travis.yml?
comment:117 follow-up: 118 Changed 3 years ago by
Suppose I delete .sage/pGroupCohomology/, to force the package to reinstall those files. Then
- if I build with Python 2 first and then Python 3, all tests pass.
- if I build with Python 3 first and then Python 2, I get failures:
[p_group_cohomology-3.3] IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/4gp2/H4gp2.sobj are not readable: expected bytes, unicode found
I guess I have a few questions:
- is it possible (and not too much work to implement) for the Python 3 version to build
.sobjfiles which are readable by Python 2? - if not, should the package discard the existing
.sobjfile and rebuild in this case? - more generally, if there is any error reading the
.sobjfile — maybe somehow the file got corrupted — should it just rebuild?
This particular problem should not happen much: people won't be switching from Python 3 to Python 2 very much. So don't work too hard to get the Python 3 version to produce .sobj files which are readable by Python 2.
comment:118 follow-up: 119 Changed 3 years ago by
Replying to jhpalmieri:
Suppose I delete
.sage/pGroupCohomology/, to force the package to reinstall those files. Then
- if I build with Python 2 first and then Python 3, all tests pass.
- if I build with Python 3 first and then Python 2, I get failures:
[p_group_cohomology-3.3] IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/4gp2/H4gp2.sobj are not readable: expected bytes, unicode found
It was a good idea to do such tests. And I am surprised about the error message: There is no use of bytes in the whole cohomology package. Perhaps the error is about unpickling a MeatAxe matrix (sage.matrix.matrix_gfpn_dense.Matrix_gfpn_dense)? But this was supposed to be fixed in #28444.
I guess I have a few questions:
- is it possible (and not too much work to implement) for the Python 3 version to build
.sobjfiles which are readable by Python 2?
I thought they were.
- if not, should the package discard the existing
.sobjfile and rebuild in this case?- more generally, if there is any error reading the
.sobjfile — maybe somehow the file got corrupted — should it just rebuild?
It is already the case that if some data source provides data in legacy format, it will automatically be overwritten by a new pickle.
This particular problem should not happen much: people won't be switching from Python 3 to Python 2 very much. So don't work too hard to get the Python 3 version to produce
.sobjfiles which are readable by Python 2.
But it should still work and needs to be investigated.
I'll follow your instructions (remove .sage/pGroupCOhomology, install (and also test?) with Python 3 and then install with Python 2 and see what is happening.
comment:119 Changed 3 years ago by
Replying to SimonKing:
I'll follow your instructions (remove
.sage/pGroupCOhomology, install (and also test?) with Python 3 and then install with Python 2 and see what is happening.
I installed and tested when I did this, but just now I hit the same error with:
- delete `.sage/pGroupCohomology'
- in Python 3, install the package
- then run Sage:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing(16, 2) H^*(SmallGroup(16,2); GF(2))
- in Python 2, install the package
- then run Sage:
sage: from pGroupCohomology import CohomologyRing sage: CohomologyRing(16, 2) --------------------------------------------------------------------------- IOError Traceback (most recent call last) <ipython-input-2-e1924d996e12> in <module>() ----> 1 CohomologyRing(Integer(16), Integer(2)) /Users/jpalmier/Desktop/Sage/sage_builds/py2/sage-8.9.rc1/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds) 1778 if q < 128: 1779 extras['websource'] = False -> 1780 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras)) 1781 return OUT 1782 /Users/jpalmier/Desktop/Sage/sage_builds/py2/sage-8.9.rc1/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds) 1209 if '@use_this_root@' in coho_options: 1210 del coho_options['@use_this_root@'] -> 1211 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_workspace,file_name), msg)) 1212 ## 3. Link with local sources and load from there 1213 elif root_local_sources and os.access(os.path.join(root_local_sources,file_name), os.R_OK) and not from_scratch: IOError: Saved data at /Users/jpalmier/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj are not readable: NEWOBJ class argument isn't a type object
comment:120 follow-up: 123 Changed 3 years ago by
Probably the following would work, too (so that it is not always needed to reinstall):
- Have the package both in py2 and py3
- Create a permanent workspace and store a cohomology ring created with py3
- Try to load the cohomology ring from that permanent workspace with py2
Anyway, after doing the py3 test suit, I can confirm what you reported: The stuff in .sage/pGroupCohomology/db/16gp2 cannot be unpickled with py2. In particular, the state descriptor cannot be loaded:
sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-2796f9b01dc5> in <module>()
----> 1 S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2845)()
157
158 ## Load file by absolute filename
--> 159 with open(filename, 'rb') as fobj:
160 X = loads(fobj.read(), compress=compress, **kwargs)
161 try:
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2795)()
158 ## Load file by absolute filename
159 with open(filename, 'rb') as fobj:
--> 160 X = loads(fobj.read(), compress=compress, **kwargs)
161 try:
162 X._default_filename = os.path.abspath(filename)
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:6929)()
1043
1044 unpickler = SageUnpickler(io.BytesIO(s), **kwargs)
-> 1045 return unpickler.load()
1046
1047
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)()
655
656 def load(self):
--> 657 return self._unpickler.load()
658
659 def noload(self):
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/structure/factory.pyx in sage.structure.factory.lookup_global (build/cythonized/sage/structure/factory.c:4615)()
748 Integer Ring
749 """
--> 750 name = bytes_to_str(name, encoding='ASCII')
751 try:
752 return factory_unpickles[name]
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/structure/factory.c:5038)()
55 return b
56 if type(b) is not bytes:
---> 57 raise TypeError(f"expected bytes, {type(b).__name__} found")
58
59 if PY_MAJOR_VERSION <= 2:
TypeError: expected bytes, unicode found
And I guess this can be fixed in bytes_to_str: After the changes in #28444, bytes_to_str is more tolerant about the input type (it may be str), but it should be even more so. It should (in py2) accept unicode.
comment:121 follow-up: 122 Changed 3 years ago by
I think it's good enough if py2-created pickles work with py3.
Require the opposite compatibility seems to be an overkill.
comment:122 Changed 3 years ago by
Replying to dimpase:
I think it's good enough if py2-created pickles work with py3.
Require the opposite compatibility seems to be an overkill.
I believe the fix (in #28540) would be relatively easy.
If you think that we shouldn't worry so much about reading py3-pickles in py2, then of course the group cohomology package can be reviewed as it is now. If I am right about #28640, then #28540 will fix the unpickling of py3-cohomology in py2 without any change in the package.
comment:123 follow-up: 124 Changed 3 years ago by
Replying to SimonKing:
And I guess this can be fixed in
bytes_to_str: After the changes in #28444,bytes_to_stris more tolerant about the input type (it may be str), but it should be even more so. It should (in py2) accept unicode.
I think that at this point it's becoming pretty clear you should use a routine that is not named bytes_to_str, but a routine "ensure_str", which in py3 accepts unicode=str and return the result unchanged, and also accepts py2's "bytes=str" objects and assumes it's "latin1" encoded, because that's what it would have been as a "str" in py2.
You are clearly NOT converting a "bytes" object to a "str" here; you are ensuring something is a "str", doing some conversions if necessary. You should use a routine that reflects that in its name if you need specific behaviour.
The reason why I care is because you are changing routines in sage.cpython.string, which people will expect to contain widely valid; well-defined routines. Stretching them up with hacks for specific purposes will lead to silent erroneous behaviour further down the line, when these routines are used in other contexts because they sound like the right tool (but aren't).
If it gets a py3 "unicode" object in py2, ideally it could just leave it as unicode, because that should be valid in str contexts in py2 for the most time as well. But we may be far from the ideal in py2. A safe choice would be str(<unicode>) to ensure you actually get a py2 str. That uses "ascii" encoding, so it may fail. You could allow for "UTF-8" but that would not round-trip correctly with the "latin1" on the other side. Using "latin1" for encoding unicode-to-str on py2 may also fail, because not all unicode code points have an encoding in "latin1".
comment:124 Changed 3 years ago by
Replying to nbruin:
The reason why I care is because you are changing routines in sage.cpython.string, which people will expect to contain widely valid; well-defined routines. Stretching them up with hacks for specific purposes will lead to silent erroneous behaviour further down the line, when these routines are used in other contexts because they sound like the right tool (but aren't).
Agreed. I have already changed the topic of #28540.
If it gets a py3 "unicode" object in py2, ideally it could just leave it as unicode, because that should be valid in str contexts in py2 for the most time as well. But we may be far from the ideal in py2. A safe choice would be
str(<unicode>)to ensure you actually get a py2 str. That uses "ascii" encoding, so it may fail.
The application (and the place where #28540 should be changed) is sage.structure.factory.lookup_global. This function used to only accept str. By #28444, it also accepts bytes, which makes it possible to unpickle a py2-pickle in py3.
I suggest in #28540 that it also accepts a unicode. This makes it possible to unpickle a py3-pickle in py2.
Again, I won't make this a dependency for the cohomology package.
comment:125 Changed 3 years ago by
#28540 brought an improvement:
sage: from pGroupCohomology import CohomologyRing
sage: H = CohomologyRing(16,2,options='debug')
_get_p_group_from_cache_or_db:
Data found at /home/king/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj
Group data are rooted at '/home/king/.sage/pGroupCohomology/db'.
Loading a cohomology ring:
The state descriptor of the to-be-unpickled ring is expected to be provided at u'/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj'
---------------------------------------------------------------------------
IOError Traceback (most recent call last)
<ipython-input-2-a62afc150e39> in <module>()
----> 1 H = CohomologyRing(Integer(16),Integer(2),options='debug')
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in __call__(self, *args, **kwds)
1778 if q < 128:
1779 extras['websource'] = False
-> 1780 OUT = self._check_compatibility(CacheKey, self._get_p_group_from_cache_or_db(GStem, KEY, **extras) or self._get_p_group_from_scratch(KEY, q, GStem, GroupName, **extras))
1781 return OUT
1782
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/pGroupCohomology/factory.pyc in _get_p_group_from_cache_or_db(self, GStem, KEY, **kwds)
1209 if '@use_this_root@' in coho_options:
1210 del coho_options['@use_this_root@']
-> 1211 raise IOError("Saved data at %s are not readable: %s"%(os.path.join(root_workspace,file_name), msg))
1212 ## 3. Link with local sources and load from there
1213 elif root_local_sources and os.access(os.path.join(root_local_sources,file_name), os.R_OK) and not from_scratch:
IOError: Saved data at /home/king/.sage/pGroupCohomology/db/16gp2/H16gp2.sobj are not readable: NEWOBJ class argument isn't a type object
So, the previous error arising in lookup_global has gone.
I am afraid the new error looks more like an error that needs to be fixed in the package, not in the Sage library. We will see...
comment:126 Changed 3 years ago by
Next step of debugging.
In Py2, where we still have py3-pickles in DOT_SAGE:
sage: from pGroupCohomology.cohomology import COHO
sage: H = COHO()
sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
sage: H.__setstate__(S)
---------------------------------------------------------------------------
IOError Traceback (most recent call last)
<ipython-input-4-9c79e2f6e0a6> in <module>()
----> 1 H.__setstate__(S)
pGroupCohomology/cohomology.pyx in pGroupCohomology.cohomology.COHO.__setstate__()
IOError: Unable to read resolution saved at /home/king/Sage/git/sage/local/share/pGroupCohomology/16gp2/dat/R16gp2.sobj
The same in py3:
sage: from pGroupCohomology.cohomology import COHO
sage: H = COHO()
sage: S = load('/home/king/.sage/pGroupCohomology/db/16gp2/dat/State.sobj')
sage: H.__setstate__(S)
sage: print(H)
Cohomology ring of Small Group number 2 of order 16 with coefficients in GF(2)
Computation complete
Minimal list of generators:
[c_2_1: 2-Cocycle in H^*(SmallGroup(16,2); GF(2)),
c_2_2: 2-Cocycle in H^*(SmallGroup(16,2); GF(2)),
c_1_0: 1-Cocycle in H^*(SmallGroup(16,2); GF(2)),
c_1_1: 1-Cocycle in H^*(SmallGroup(16,2); GF(2))]
Minimal list of algebraic relations:
[c_1_0^2,
c_1_1^2]
So, the error is now moved to __setstate__. I guess I'll fix it there.
comment:127 follow-up: 128 Changed 3 years ago by
Meanwhile I know at what point the error mentioned in comment:125 arises:
sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
---------------------------------------------------------------------------
UnpicklingError Traceback (most recent call last)
<ipython-input-2-855d0a850e70> in <module>()
----> 1 si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/explain_pickle.pyc in unpickle_newobj(klass, args)
2438 return pers[int(id)]
2439
-> 2440 return SageUnpickler.loads(pickle, persistent_load=pers_load)
2441
2442
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.SageUnpickler.loads (build/cythonized/sage/misc/persist.c:6467)()
962 """
963
--> 964 return cls(io.BytesIO(data), **kwargs).load()
965
966
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)()
655
656 def load(self):
--> 657 return self._unpickler.load()
658
659 def noload(self):
UnpicklingError: NEWOBJ class argument isn't a type object
But I do not have the faintest idea yet what this error means. For comparison, in py3 I get
sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
sage: si1
<pGroupCohomology.resolution.RESL_sparse_unpickle_class object at 0x7f2ee987abe0>
I'm guessing that a class in Python-3 is a type, but in Python-2 it generally isn't. Is that the meaning of the error? But isn't it the case that a class defined in Cython (here: resolution.pyx) automatically is a type?
Anyway, I'll try to turn class RESL_sparse_unpickle_class into class RESL_sparse_unpickle_class(object) and see if it helps.
comment:128 Changed 3 years ago by
Replying to SimonKing:
Anyway, I'll try to turn
class RESL_sparse_unpickle_classintoclass RESL_sparse_unpickle_class(object)and see if it helps.
Nope.
sage: from pGroupCohomology.resolution import RESL_sparse_unpickle_class sage: RESL_sparse_unpickle_class <class 'pGroupCohomology.resolution.RESL_sparse_unpickle_class'> sage: isinstance(_, type) True
but still
sage: pg_RESL_sparse_unpickle_class = unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class')
sage: si1 = unpickle_newobj(pg_RESL_sparse_unpickle_class, ())
Traceback (most recent call last):
...
UnpicklingError: NEWOBJ class argument isn't a type object
comment:129 Changed 3 years ago by
Odd. After trying again, the error is gone. In Py2:
sage: R = load("/home/king/.sage/pGroupCohomology/db/16gp2/dat/R16gp2.sobj")
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-1-c703eea54ef8> in <module>()
----> 1 R = load("/home/king/.sage/pGroupCohomology/db/16gp2/dat/R16gp2.sobj")
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2845)()
157
158 ## Load file by absolute filename
--> 159 with open(filename, 'rb') as fobj:
160 X = loads(fobj.read(), compress=compress, **kwargs)
161 try:
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.load (build/cythonized/sage/misc/persist.c:2795)()
158 ## Load file by absolute filename
159 with open(filename, 'rb') as fobj:
--> 160 X = loads(fobj.read(), compress=compress, **kwargs)
161 try:
162 X._default_filename = os.path.abspath(filename)
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:6929)()
1043
1044 unpickler = SageUnpickler(io.BytesIO(s), **kwargs)
-> 1045 return unpickler.load()
1046
1047
/home/king/Sage/git/sage/local/lib/python2.7/site-packages/sage/misc/persist.pyx in sage.misc.persist._BaseUnpickler.load (build/cythonized/sage/misc/persist.c:5806)()
655
656 def load(self):
--> 657 return self._unpickler.load()
658
659 def noload(self):
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL_sparse_unpickle_class.__call__()
pGroupCohomology/resolution.pyx in pGroupCohomology.resolution.RESL.__init__()
OSError: Bad file format in file pgroup.c (line 404)
So, now complain about NEWOBJ any more.
comment:130 Changed 3 years ago by
Spoke too soon. Immediately after this different error (i.e., in the same session), I again got the "NEWOBJ" error.
comment:131 Changed 3 years ago by
sage: from pGroupCohomology.resolution import RESL_sparse_unpickle_class sage: pg_RESL_sparse_unpickle_class is RESL_sparse_unpickle_class False
In a different session, I got the answer "True".
Anyway, the error about "NEWOBJ" comes from python. It seems that python decided to kill data persistency. I tried to work around to the extent that py2-pickles of cohomology rings seem to be legible in py3. But I think I will not work further on making py3-pickles legible in py2.
Another question: Why is the branch of this ticket shown red?
comment:132 follow-up: 133 Changed 3 years ago by
The oddity is: Sometimes pg_RESL_sparce_unpickle_class is RESL_sparse_unpickle_class and sometimes it isn't. When the error occurs, I get
sage: pg_RESL_sparse_unpickle_class <class pGroupCohomology.auxiliaries.unpickle_old_resl at 0x7fb8d4f549a8>
It is not nice that unpickle_global('pGroupCohomology.resolution', 'RESL_sparse_unpickle_class') sometimes returns something else. But perhaps it is worth trying to make this class inherit from object.
comment:133 Changed 3 years ago by
Replying to SimonKing:
But perhaps it is worth trying to make this class inherit from
object.
Nope, didn't help. As there doesn't seem to be an easy fix, I do not plan to work further on it.
Do I understand correctly that I have to rebase on the current develop branch?
comment:134 Changed 3 years ago by
| Commit: | 463806c6cebb3120e20df0f19a216dd529553741 → 7ad6a4a35c9af94d1692a17e1b8855f49d60a38e |
|---|
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
| 7ad6a4a | Upgrade p_group_cohomology to version 3.3
|
comment:135 Changed 3 years ago by
Maybe it would be simpler, if any error arises reading a pickle, to just delete and reproduce the pickle file. Compatibility going from Python 3 to Python 2 may be annoying to debug and to fix, and besides, there may be other errors that we haven't foreseen.
comment:136 Changed 3 years ago by
Or as you say, to just not work on the Py3 -> Py2 conversion. That's fine with me.
comment:137 Changed 3 years ago by
I am using unpickle_override for RESL_sparse_unpickle_class, but I am not sure if that's needed at all.
comment:138 Changed 3 years ago by
Bummer. Apparently I have destroyed the git repository for my py3 installation. It says:
king@klap:~/Sage/git/py3$ git lg fatal: your current branch 't/28444/fix_backwards_incompatibility_of_unpickling_in_python_3' does not have any commits yet
and ALL files in the Sage tree are untracked. How has that happened, and how to fix it?
comment:139 follow-up: 142 Changed 3 years ago by
Hopefully John has a version of this branch just before you did that fateful forced push.
There could be copies left on patchbots, maybe LinuxMint?/18.2/x86_64/4.8.0-53-generic/pc78-math patchbot does have it...
comment:140 Changed 3 years ago by
Unfortunately I don't, but there are the commits in comment:113 which are still accessible.
comment:141 Changed 3 years ago by
yes, indeed, these commits are available in the full "git fetch trac" - I have them locally, in case they get gc'd.
comment:142 Changed 3 years ago by
Replying to dimpase:
Hopefully John has a version of this branch just before you did that fateful forced push.
- What's wrong with the forced push? IMHO, that has nothing to to with the problem whatsoever. The forced push is nothing more than a rebasing of the previously committed branch. After pushing it, I could still work.
- The branch is still present in my git repository. Perhaps I didn't make it clear. I have a single git repository with two worktrees, one for py2, one for py3. The py2 worktree is fine (AFAIC), in particular I can checkout any branch. However, it seems that I deleted the branch on which the py3 worktree was living, with
git branch -d. As a result, all files in the py3 worktree are now untracked, as is shown bygit status, andgit logsays that there has never been a commit. Again, I am talking about only one of my two worktrees. - The deleted branch is the branch from #28444, not from here. That branch is merged in "develop", that's why I did
git branch -don it.
So, from my perspective, "solving the problem" means "allowing me to do git checkout <branchname> in the py3-worktree" -- this currently is refused, because git claims that I should first commit or stash the whole Sage library.
comment:143 Changed 3 years ago by
Would git checkout -f <branch_name> possibly help? Or would I possibly destroy even more?
comment:144 follow-up: 145 Changed 3 years ago by
You could try git stash and then git checkout BRANCH and then git stash COMMAND for some value of COMMAND: git stash pop? git stash apply? git stash branch NEW?
comment:145 Changed 3 years ago by
Replying to jhpalmieri:
You could try
git stashand thengit checkout BRANCHand thengit stash COMMANDfor some value ofCOMMAND:git stash pop?git stash apply?git stash branch NEW?
Certainly not. I do in fact not have any uncommitted changes! Instead, git believes that I have uncommitted changes (indeed it believes that the whole sage library is completely new), because git is convinced that I am on a branch with an empty commit history -- namely, it has allowed me to delete the branch on which I was checked out.
comment:146 follow-up: 147 Changed 3 years ago by
You could first also make a copy of the whole directory, in case git stash etc. screws things up even more. How much can you recover by doing diff -r /path/to/py2/worktree /path/to/py3/worktree, or maybe git diff /path/to/py2/worktree /path/to/py3/worktree?
comment:147 follow-up: 148 Changed 3 years ago by
Replying to jhpalmieri:
You could first also make a copy of the whole directory,
Hopefully the other worktree (that is still working!) is the whole directory.
in case
git stashetc. screws things up even more. How much can you recover by doingdiff -r /path/to/py2/worktree /path/to/py3/worktree, or maybegit diff /path/to/py2/worktree /path/to/py3/worktree?
That would be a mega-patch. It would be the diff of an empty folder and the complete Sage sources.
comment:148 follow-up: 149 Changed 3 years ago by
Replying to SimonKing:
Replying to jhpalmieri:
You could first also make a copy of the whole directory,
Hopefully the other worktree (that is still working!) is the whole directory.
in case
git stashetc. screws things up even more. How much can you recover by doingdiff -r /path/to/py2/worktree /path/to/py3/worktree, or maybegit diff /path/to/py2/worktree /path/to/py3/worktree?That would be a mega-patch. It would be the diff of an empty folder and the complete Sage sources.
Why? diff -r ... should just compare the files, and the files exist in both places, regardless of whether they're tracked by git.
By the way, if the branch still exists, can you create a new workspace using that branch?
comment:149 Changed 3 years ago by
Replying to jhpalmieri:
Why?
diff -r ...should just compare the files, and the files exist in both places, regardless of whether they're tracked by git.
I see. Then the diff is empty.
By the way, if the branch still exists, can you create a new workspace using that branch?
That's my question. The branch for this ticket still exists, in fact it is the branch that is currently checked out in my py2 worktree. However, git won't allow me to do any normal (non-forced) operation on my py3 worktree, as the branch that was checked out there was deleted, thus, git assumes that my py3 worktree has no commit history and has a couple of thousands new files.
comment:150 Changed 3 years ago by
So if the two worktrees aren't any different from the content point of view, maybe you should just create an entirely new worktree, in a new directory, using the existing branch, and delete the old one. It means you will have to rebuild Sage in the new worktree. I guess that's what you're trying to avoid?
Alternatively, you could rename the old directory (OLD -> TEMP), create a new worktree where the old one used to be (OLD). Then in the shell, but without telling git, you could delete that newly created directory and rename TEMP back to OLD. Then see what git says. I think git keeps track of its worktree information in the original folder, not in the newly created one. By the way, is there anything in .git/worktrees/? If so, you might need to remove that before you go through this procedure.
comment:151 follow-up: 153 Changed 3 years ago by
I'm getting a few doctest failures with a Python 3 build of Sage, version 9.0.beta0. All failures have the form
File "pGroupCohomology-3.3/pGroupCohomology/cohomology.pyx", line 4591, in pGroupCohomology.cohomology.COHO.__dir__
Failed example:
[a for a in dir(H) if a.startswith('f')] #indirect doctest
Expected:
['filter_degree_type',
'filter_regular_gready_parameters',
'filter_regular_parameters',
'find_dickson',
'find_dickson_in_subgroup',
'find_small_last_parameter',
'firstOdd',
'fraction_field',
'from_base_ring']
Got:
['filter_degree_type',
'filter_regular_gready_parameters',
'filter_regular_parameters',
'find_dickson',
'find_dickson_in_subgroup',
'find_small_last_parameter',
'firstOdd',
'fraction_field',
'free_module',
'from_base_ring']
comment:153 Changed 3 years ago by
Replying to jhpalmieri:
'free_module',
So, if I understand correctly, the problem is both in py2 and py3., and it is about an additional entry in the dir. That should be solvable...
comment:154 follow-up: 155 Changed 3 years ago by
Is that kind of doctest too fragile? How important is it? You could certainly change it to startswith('fi') or add 'free_module' to the list, but maybe there are more robust options. 'fraction_field' in dir(H)? I guess I don't know what it's supposed to be testing.
comment:155 Changed 3 years ago by
Replying to jhpalmieri:
Is that kind of doctest too fragile? How important is it? You could certainly change it to
startswith('fi')or add'free_module'to the list, but maybe there are more robust options.'fraction_field' in dir(H)? I guess I don't know what it's supposed to be testing.
It is supposed to test introspection and autocompletion (which doesn't work in Py-3, if I understand correctly).
On the other hand, it is true that all methods in that list that do not start with "fi" are inherited and are thus easily changed. Perhaps it is best to insert an ellipsis.
comment:156 Changed 3 years ago by
| Commit: | 7ad6a4a35c9af94d1692a17e1b8855f49d60a38e → c247f13fd6e6ae3d3feeef41ebba1a78346744da |
|---|
Branch pushed to git repo; I updated commit sha1. New commits:
| c247f13 | Fix some introspection tests
|
comment:157 Changed 3 years ago by
comment:158 Changed 3 years ago by
Hooray, with some long tests:
$ ./sage -f -c p_group_cohomology ... [p_group_cohomology-3.3] ---------------------------------------------------------------------- [p_group_cohomology-3.3] All tests passed! [p_group_cohomology-3.3] ---------------------------------------------------------------------- ... [sagelib-8.9.rc1] real 0m18.161s [sagelib-8.9.rc1] user 0m5.983s [sagelib-8.9.rc1] sys 0m1.083s make[2]: Leaving directory '/home/king/Sage/git/py3/build/make' make[1]: Leaving directory '/home/king/Sage/git/py3/build/make' real 0m18.438s user 0m6.252s sys 0m1.126s Sage build/upgrade complete!
So, it seems to work!
comment:159 Changed 3 years ago by
I still get one failure:
File "pGroupCohomology-3.3/pGroupCohomology/cohomology.pyx", line 4814, in pGroupCohomology.cohomology.COHO.__getattr__
Failed example:
tc.completions('H.f',globals())
Expected:
['H.filter_degree_type',
'H.filter_regular_gready_parameters',
'H.filter_regular_parameters',
'H.find_dickson',
'H.find_dickson_in_subgroup',
'H.find_small_last_parameter',
'H.firstOdd',
'H.fraction_field',
'H.from_base_ring']
Got:
['H.filter_degree_type',
'H.filter_regular_gready_parameters',
'H.filter_regular_parameters',
'H.find_dickson',
'H.find_dickson_in_subgroup',
'H.find_small_last_parameter',
'H.firstOdd',
'H.fraction_field',
'H.free_module',
'H.from_base_ring']
comment:160 Changed 3 years ago by
Too bad that those tests work for me, i.e., it is easy to miss one of those tests.
Trying again (in a few minutes...).
comment:161 Changed 3 years ago by
| Commit: | c247f13fd6e6ae3d3feeef41ebba1a78346744da → 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 |
|---|
Branch pushed to git repo; I updated commit sha1. New commits:
| 6c5dafc | Silence another flaky doctest
|
comment:162 Changed 3 years ago by
comment:163 Changed 3 years ago by
| Reviewers: | → John Palmieri |
|---|---|
| Status: | needs_review → positive_review |
All tests passed!
comment:164 Changed 3 years ago by
| Milestone: | sage-8.9 → sage-9.0 |
|---|
comment:165 Changed 3 years ago by
| Branch: | u/SimonKing/p_group_cohomology_in_py_3 → 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 |
|---|---|
| Resolution: | → fixed |
| Status: | positive_review → closed |
comment:166 Changed 3 years ago by
| Commit: | 6c5dafcd777ae5b36b22b4e2fd1bb242d7af5cd0 |
|---|
sdh_X scripts call sdh_die themselves, so doing
sdh_X || sdh_die ... is equivalent to simply sdh_X.
The intention of calling sdh_die to produce a meaninful error message is commendable, but these after || just won't be called.
It's better to modify sdh_X to allow for custiom error messages, I suppose.
cmpis also not supported in Python 3, and for a dictionaryd, you should dox in drather thand.has_key(x).