Opened 3 months ago
Closed 6 weeks ago
#30402 closed defect (fixed)
Cannot load an object containing a matrix if it was saved from Python 2 Sage
Reported by:  ghpenguian  Owned by:  

Priority:  blocker  Milestone:  sage9.2 
Component:  pickling  Keywords:  load, save, matrix 
Cc:  nbruin, jhpalmieri, chapoton, slelievre  Merged in:  
Authors:  Nils Bruin, John Palmieri, Jonathan Kliem  Reviewers:  Dima Pasechnik, Paul Leopardi 
Report Upstream:  N/A  Work issues:  
Branch:  c352b8a (Commits)  Commit:  c352b8ae5c59a3113ef35c888ca6118a322a6f45 
Dependencies:  Stopgaps: 
Description (last modified by )
I am currently trying to convert my BooleanCayleygraphs project from Python 2 to Python 3 Sage. https://github.com/penguian/BooleanCayleygraphs/tree/23porttopython3
I have succeeded in converting the code, but cannot load objects that were saved by my previous Python 2based code. These objects contain matrices as members.
As far as I can tell, load(...,encoding='latin1') fails to load a dict containing a matrix but load(...,encoding='bytes') loads but does not restore the original object. See the attachments, showing a script run on Cocalc, and its results.
Along the way, we fix some code style etc. in `src/sage/matrix/matrix_integer_dense.pyx'.b
Blocker for 9.2 because this defect is in the way of adoption of python3based Sage versions by users.
Attachments (2)
Change History (23)
Changed 3 months ago by
comment:1 Changed 3 months ago by
 Description modified (diff)
comment:2 Changed 3 months ago by
comment:3 Changed 3 months ago by
I applied the change suggested by Nils Bruin to Sage 9.1 installed from source on Kubuntu 20.04, and it works for me.
comment:4 Changed 3 months ago by
 Cc nbruin jhpalmieri added
 Priority changed from major to critical
Could someone please prepare a branch for this ticket?
comment:5 Changed 3 months ago by
 Cc chapoton slelievre added
comment:6 Changed 7 weeks ago by
It would be really nice to get this fix in.
comment:7 Changed 7 weeks ago by
I have a branch, but I am having problems pushing it to trac. If someone else wants to prepare it, go ahead.

src/sage/matrix/matrix_integer_dense.pyx
diff git a/src/sage/matrix/matrix_integer_dense.pyx b/src/sage/matrix/matrix_integer_dense.pyx index fa6971504b..98aee236eb 100644
a b cdef class Matrix_integer_dense(Matrix_dense): 550 550 return data 551 551 552 552 def _unpickle(self, data, int version): 553 """ 554 TESTS:: 555 556 sage: b = matrix(ZZ,2,3, [0,0,0, 0, 0, 0]) 557 sage: s = b'1 61 f 2 3 0' 558 sage: t = s.decode() 559 sage: b._unpickle(s, 0) == b._unpickle(t, 0) 560 True 561 sage: b 562 [ 1 193 15] 563 [ 2 3 0] 564 """ 553 565 if version == 0: 566 if isinstance(data, str): 567 # old Py2 pickle: old "bytes" object reaches us as a 568 # latin1encoded string. 569 data = data.encode('latin1') 554 570 if isinstance(data, bytes): 555 571 self._unpickle_version0(data) 556 572 elif isinstance(data, list):
Needs testing.
comment:8 Changed 7 weeks ago by
I prepared it and I have the same problem, it seems:
kliem@cofio:~/localhome/sage/src/sage/matrix$ git push setupstream trac public/30402 perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_PAPER = "de_DE.UTF8", LC_ADDRESS = "de_DE.UTF8", LC_MONETARY = "de_DE.UTF8", LC_NUMERIC = "de_DE.UTF8", LC_TELEPHONE = "de_DE.UTF8", LC_IDENTIFICATION = "de_DE.UTF8", LC_MEASUREMENT = "de_DE.UTF8", LC_TIME = "de_DE.UTF8", LC_NAME = "de_DE.UTF8", LANG = "en_US.UTF8" are supported and installed on your system. perl: warning: Falling back to the standard locale ("C"). Enumerating objects: 35, done. Counting objects: 100% (35/35), done. Delta compression using up to 8 threads Compressing objects: 100% (30/30), done. Writing objects: 100% (30/30), 3.57 KiB  3.57 MiB/s, done. Total 30 (delta 25), reused 0 (delta 0) remote: error: insufficient permission for adding an object to repository database ./objects remote: fatal: failed to write object error: remote unpack failed: unpackobjects abnormal exit To trac.sagemath.org:sage.git ! [remote rejected] public/30402 > public/30402 (unpacker error) error: failed to push some refs to 'git@trac.sagemath.org:sage.git'
comment:9 Changed 7 weeks ago by
please try again now. I've been doing some maintainance/repairing of the git repo.
comment:10 Changed 7 weeks ago by
same error
comment:11 Changed 7 weeks ago by
OK, let me see.
comment:12 Changed 7 weeks ago by
 Branch set to public/ticket/30402py23
 Commit set to 342030c403a30a41b87c7efbf977d79bf1ca4afb
 Status changed from new to needs_review
I've fixed the permissions on ...repository database ./objects
and was able to push the branch.
There seems to be some problem there caused by gitolite  the thing that enables viewing branches here in the browser, which also uses the repo on the server. Weird.
New commits:
342030c  fix (un)pickle

comment:13 Changed 7 weeks ago by
 Reviewers set to Dima Pasechnik, Paul Leopardi
 Status changed from needs_review to positive_review
ok, the test passes. Good to go.
comment:14 Changed 7 weeks ago by
I fixed some codestyle and stuff. Should I push it or put it on a new ticket.
comment:15 Changed 7 weeks ago by
push it here if you like, why not.
comment:16 Changed 7 weeks ago by
 Commit changed from 342030c403a30a41b87c7efbf977d79bf1ca4afb to c352b8ae5c59a3113ef35c888ca6118a322a6f45
 Status changed from positive_review to needs_review
comment:17 Changed 7 weeks ago by
Ok.
Maybe one day sage tox
for all of sage :)
It doesn't even pass this file yet due to coverage and it not liking the variable name ans
.
comment:18 Changed 7 weeks ago by
 Status changed from needs_review to positive_review
OK, great, thanks. Please add yourself to reviewers/authors, if you like.
comment:19 Changed 7 weeks ago by
 Description modified (diff)
comment:20 Changed 7 weeks ago by
 Description modified (diff)
 Priority changed from critical to blocker
comment:21 Changed 6 weeks ago by
 Branch changed from public/ticket/30402py23 to c352b8ae5c59a3113ef35c888ca6118a322a6f45
 Resolution set to fixed
 Status changed from positive_review to closed
Test script to run on Cocalc