Cannot load an object containing a matrix if it was saved from Python 2 Sage

[gh-penguian]

I am currently trying to convert my Boolean-Cayley-graphs project from Python 2 to Python 3 Sage. ​https://github.com/penguian/Boolean-Cayley-graphs/tree/23-port-to-python-3

I have succeeded in converting the code, but cannot load objects that were saved by my previous Python 2-based code. These objects contain matrices as members.

As far as I can tell, load(...,encoding='latin-1') fails to load a dict containing a matrix but load(...,encoding='bytes') loads but does not restore the original object. See the attachments, showing a script run on Cocalc, and its results.

Along the way, we fix some code style etc. in `src/sage/matrix/matrix_integer_dense.pyx'.b

Blocker for 9.2 because this defect is in the way of adoption of python3-based Sage versions by users.

[gh-penguian]

Test script to run on Cocalc

[gh-penguian]

Results of running test script with different versions of Sage

[gh-penguian]

See also ​https://groups.google.com/g/sage-support/c/3WKui-LojKs/m/DgEPBGF3AQAJ and ​https://trac.sagemath.org/ticket/28444

[gh-penguian]

I applied the ​change suggested by Nils Bruin to Sage 9.1 installed from source on Kubuntu 20.04, and it works for me.

[mkoeppe]

Could someone please prepare a branch for this ticket?

[mkoeppe]

It would be really nice to get this fix in.

[jhpalmieri]

I have a branch, but I am having problems pushing it to trac. If someone else wants to prepare it, go ahead.

src/sage/matrix/matrix_integer_dense.pyx

@@ -550 +550 @@
         return data
 
     def _unpickle(self, data, int version):
+        """
+        TESTS::
+
+            sage: b = matrix(ZZ,2,3, [0,0,0, 0, 0, 0])
+            sage: s = b'1 61 f -2 3 0'
+            sage: t = s.decode()
+            sage: b._unpickle(s, 0) == b._unpickle(t, 0)
+            True
+            sage: b
+            [  1 193  15]
+            [ -2   3   0]
+        """
         if version == 0:
+            if isinstance(data, str):
+                # old Py2 pickle: old "bytes" object reaches us as a
+                # latin1-encoded string.
+                data = data.encode('latin1')
             if isinstance(data, bytes):
                 self._unpickle_version0(data)
             elif isinstance(data, list):

Needs testing.

[gh-kliem]

I prepared it and I have the same problem, it seems:

kliem@cofio:~/localhome/sage/src/sage/matrix$ git push --set-upstream trac public/30402

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_PAPER = "de_DE.UTF-8",
	LC_ADDRESS = "de_DE.UTF-8",
	LC_MONETARY = "de_DE.UTF-8",
	LC_NUMERIC = "de_DE.UTF-8",
	LC_TELEPHONE = "de_DE.UTF-8",
	LC_IDENTIFICATION = "de_DE.UTF-8",
	LC_MEASUREMENT = "de_DE.UTF-8",
	LC_TIME = "de_DE.UTF-8",
	LC_NAME = "de_DE.UTF-8",
	LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Enumerating objects: 35, done.
Counting objects: 100% (35/35), done.
Delta compression using up to 8 threads
Compressing objects: 100% (30/30), done.
Writing objects: 100% (30/30), 3.57 KiB | 3.57 MiB/s, done.
Total 30 (delta 25), reused 0 (delta 0)
remote: error: insufficient permission for adding an object to repository database ./objects
remote: fatal: failed to write object
error: remote unpack failed: unpack-objects abnormal exit
To trac.sagemath.org:sage.git
 ! [remote rejected]       public/30402 -> public/30402 (unpacker error)
error: failed to push some refs to 'git@trac.sagemath.org:sage.git'

[dimpase]

please try again now. I've been doing some maintainance/repairing of the git repo.

[gh-kliem]

same error

[dimpase]

OK, let me see.

[dimpase]

I've fixed the permissions on ...repository database ./objects and was able to push the branch.

There seems to be some problem there caused by gitolite - the thing that enables viewing branches here in the browser, which also uses the repo on the server. Weird.

---

New commits:

​342030c

fix (un)pickle

[dimpase]

ok, the test passes. Good to go.

[gh-kliem]

I fixed some codestyle and stuff. Should I push it or put it on a new ticket.

[dimpase]

push it here if you like, why not.

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​d798a42	fix codestyle etc.
​44e5151	pycodestyle
​c352b8a	added indirect doctest flag

[gh-kliem]

Ok.

Maybe one day sage -tox for all of sage :-)

It doesn't even pass this file yet due to coverage and it not liking the variable name ans.

[dimpase]

OK, great, thanks. Please add yourself to reviewers/authors, if you like.