sage.libs.ecl: Fix unicode handling

[mkoeppe]

As a follow-up to #29278, #29280: If we use Unicode variable names in SR, declaring a domain gives an error:

sage: SR.var('π', domain='real')
RuntimeError: ECL says: THROW: The catch MACSYMA-QUIT is undefined.
SystemError: <built-in method var of sage.symbolic.ring.SymbolicRing object at 0x334506908> returned a result with an error set

This comes from our ECL interface:

sage: from sage.libs.ecl import *
sage: u_symbol = EclObject('🔥')
sage: u_symbol
<repr(<sage.libs.ecl.EclObject at 0x337e7b3c8>) failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 2: invalid start byte>
sage: u_symbol.python()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 2: invalid start byte

Also note:

sage: b_symbol = EclObject(bytes([166]))
sage: b_symbol
<repr(<sage.libs.ecl.EclObject at 0x337e7b058>) failed: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa6 in position 0: invalid start byte>

[mkoeppe]

Not sure if there is a function in the ECL C API that constructs a Unicode Lisp string. So I would just be going through CODE-CHAR as in

sage: ecl_eval('''(type-of (map 'string #'code-char '(128293))))''')
<ECL: (SIMPLE-ARRAY CHARACTER (1))>

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b6ebb52	src/sage/cpython/*string*: Update documentation
​6ce8f9e	python_to_ecl: Handle unicode strings

[mkoeppe]

Now

sage: SR.var('π', domain='real')
π

but more work is needed.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c983523

EclObject.str: Handle unicode

[mkoeppe]

sage: from sage.libs.ecl import *
sage: u_symbol = EclObject('🔥')
sage: u_symbol
<ECL: 🔥>

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f237935

ecl_to_python: Handle unicode strings

[mkoeppe]

sage: u_string = EclObject('"🐙"')
sage: u_string
<ECL: "🐙">
sage: _.python()
'"🐙"'

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6f740eb

ecl_eval: Handle unicode strings

[mkoeppe]

sage: clock = ecl_eval('''(defun clock (h) (string (elt "🕛🕐🕑🕒🕓🕔🕕🕖🕗🕘🕙🕚" (mod h 12))))''')
sage: clock(3).python()
'"🕒"'

[gh-mwageringel]

Thank you for solving this.

Should our Maxima interface support unicode with this branch now, or do you know what else is needed to make this work? The following used to give an error, but results in a crash now:

sage: var('ξ')._maxima_()

Condition of type: SIMPLE-ERROR
Cannot coerce string Cannot coerce string $_SAGE_VAR_ξ to a base-string to a base-string
No restarts available.
...
Excessive debugger depth! Probable infinite recursion!
Quitting.

Also, please add a doctest for the new functionality.

[dimpase]

Are you on ecl 20.4 ?

[mkoeppe]

Replying to gh-mwageringel:

The following used to give an error, but results in a crash now:

sage: var('ξ')._maxima_()

Condition of type: SIMPLE-ERROR
Cannot coerce string Cannot coerce string $_SAGE_VAR_ξ to a base-string to a base-string
No restarts available.
...
Excessive debugger depth! Probable infinite recursion!
Quitting.

Yes, I can confirm this here. I'll look into this.

Also, please add a doctest for the new functionality.

Sure, will do.

[mkoeppe]

Replying to dimpase:

Are you on ecl 20.4 ?

I'm still on 16.1.2.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ee97a6d

Fix infinite recursion when error messages are not base-strings

[dimpase]

I suggest we work on ecl 20.4, for uniformity

[mkoeppe]

The above error is due to a bug in ECL 16.1.2:

> (princ-to-string 'ξ)
"Ξ"
> (setf local-table (copy-readtable nil))
> (setf (readtable-case local-table) :invert)
> :INVERT
> (let ((*readtable* local-table) (*print-case* :upcase)) (princ-to-string 'ξ))
Cannot coerce string Ξ to a base-string

I will try with the ECL upgrade now.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​0076bc3	backport Maxima fix for bug #3629
​1d074e8	doctest fixes
​266d8c1	backport ECL PR #210
​12447bc	reject old makeinfo
​a3e0eca	add upstream fix from MR 215
​89b006b	add the patch from upstream MR 214
​0b77737	add upstream MR 216 (to fix cygwin fork)
​8ca1c0e	Merge tag '9.2.beta2' into t/22191/public/packages/ecl20
​f82c716	Commit 75877dd8 from upstream
​2d87ee3	Merge commit 'f82c716fdf9c6e91a07166d36b6329a15ecfb41d' of git://trac.sagemath.org/sage into t/30106/sage_libs_ecl__fix_unicode_handling

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​828a727

Add doctests

[mkoeppe]

Replying to mkoeppe:

The above error is due to a bug in ECL 16.1.2:

> (princ-to-string 'ξ)
"Ξ"
> (setf local-table (copy-readtable nil))
> (setf (readtable-case local-table) :invert)
> :INVERT
> (let ((*readtable* local-table) (*print-case* :upcase)) (princ-to-string 'ξ))
Cannot coerce string Ξ to a base-string

I will try with the ECL upgrade now.

This bug is still present in ECL 20.4.24. The above code is from Maxima's PRINT-INVERT-CASE function in commac.lisp.

[mkoeppe]

Note that after commit ee97a6d, the test case from comment 17 now gives a proper error message, no longer a crash:

sage: var('ξ')._maxima_()
....: 
<repr(<sage.interfaces.maxima_lib.MaximaLibElement at 0x32c1d4828>) failed: RuntimeError: ECL says: Cannot coerce string $_SAGE_VAR_ξ to a base-string>

Fixing this ECL bug or working around it is outside of the scope of this ticket.

[gh-mwageringel]

Ok, thanks for the fix. There is one more place where ecl_string_to_python should be used – in print_objects:

sage: from sage.libs.ecl import *
sage: u_symbol = EclObject('🔥')
sage: print_objects()  # crashes

Moreover, the docstring of ecl_eval should be changed into a raw string, as the examples contain backslashes.

Once that is fixed, you can set a positive review on my behalf.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​99b894a	print_objects: Handle unicode
​59dd62b	ecl_eval: Make docstring raw

[mkoeppe]

Thank you!

[mkoeppe]

Follow-up ticket to keep track of the ECL issue: #30122