Python 3 preparation: Handle basestring (Py2) vs. str (Py3)

[Wilfried Luebbe]

From the Python 2 documentation ​basestring():

This abstract type is the superclass for str and unicode. It cannot be called or instantiated, but it can be used to test whether an object is an instance of str or unicode. isinstance(obj, basestring) is equivalent to isinstance(obj, (str, unicode)). New in version 2.3. This is not available in Python 3.

The tool 2to3 changes basestring into str (the only string type in Py3).

There are 27 effected modules.

This ticket is tracked as a dependency of meta-ticket ticket:16052.

[aapitzsch]

New commits:

​e1252a4

replace basestring by str

[Ralf Stephan]

Eyeballed and patchbotted ok. Just a naive question: wouldn't the ... str = basestring snippet be needed in every affected file?

[Wilfried Luebbe]

For unicode strings in Python 2.7 the change will not give the same result as before. See

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> s ='a str'
>>> su = u'a u-str'
>>> isinstance(s, basestring)
True
>>> isinstance(su, basestring)
True
>>> isinstance(s, str)
True
>>> isinstance(su, str)
False
>>> 

As we want a single Python 2 / 3 code base I think this is not the right solution.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f4ce4d1	replace basestring by str if python version >= 3
​7fc09f9	Merge remote-tracking branch 'origin/develop' into py3_basestring

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​63eaffd

Merge remote-tracking branch 'origin/develop' into py3_basestring

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3df268b

Merge remote-tracking branch 'origin/develop' into py3_basestring

[Jeroen Demeyer]

I have a general question about Python3 tickets: would it not be better to simply use the 2to3 tool at build time (i.e. when doing sage -b)? This would fix not only this ticket, but also many others and there would be no need to uglify the code like in this patch.

[aapitzsch]

You are right using 2to3 would be the better way, but the tool doesn't touch doctests and it will not fix all compatibility issues.

IMO we should change as much as possible (if it is not ugly) upstream to reduce work for 2to3. And in the long run we have to change it anyway. Changes like #16442, #16529 and even #16071 (if we leave the izip part out) should be fine.

Let's skip this ticket for the moment and maybe replaced it by using 2to3 at build time.

[Jeroen Demeyer]

Cython files should support basestring out of the box in Python 3. So no changes are needed to .pyx files.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bc70272

revert basestring changes in *.pyx files

[Marc Mezzarobba]

You wrote:

Replying to aapitzsch:

Let's skip this ticket for the moment and maybe replaced it by using 2to3 at build time.

but later set the ticket to needs_review... So I don't understand, what is the status of this ticket? I'm changing both the milestone to pending and the status to needs_work (as there are merge conflicts anyway), please feel free to revert one or both if appropriate.

[Jeroen Demeyer]

See #18492 for a much better approach using six.

[aapitzsch]

Made use of six.

I haven't added six as a dependency because that's done in #18492.

[Wilfried Luebbe]

The patch looks fine. Testing is OK.

But in sage-6.8.beta3 there are still basestrings in sage/quivers/algebra.py line 277 and in sage/quivers/path_semigroup.py lines 340 / 348. The are NOT present sage-6.8.beta0 on which the patch is based. I haven't found out why.

Can you base the patch on sage-6.8.beta3 and change the 3 remaining basestrings. Then I would set the ticket to positive review.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eaa8dc7	Merge remote-tracking branch 'origin/develop' into py3_basestring_six
​b16df45	replace remaining basestring by six.string_types

[Wilfried Luebbe]

Right at the beginning of test --all --long (after a fresh make!) I get LOTS of errors like

File "src/sage/combinat/root_system/non_symmetric_macdonald_polynomials.py", line 62, in sage.combinat.root_system.non_symmetric_macdonald_polynomials.NonSymmetricMacdonaldPolynomials
Failed example:
    E = NonSymmetricMacdonaldPolynomials(["A",2,1])
Exception raised:
    Traceback (most recent call last):
      File "/home/wluebbe/sage-6.8.beta3/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 496, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/wluebbe/sage-6.8.beta3/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 858, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.root_system.non_symmetric_macdonald_polynomials.NonSymmetricMacdonaldPolynomials[0]>", line 1, in <module>
        E = NonSymmetricMacdonaldPolynomials(["A",Integer(2),Integer(1)])
      File "sage/misc/lazy_import.pyx", line 383, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3453)
        return self._get_object()(*args, **kwds)
      File "sage/misc/classcall_metaclass.pyx", line 326, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1204)
        return cls.classcall(cls, *args, **kwds)
      File "/home/wluebbe/sage-6.8.beta3/local/lib/python2.7/site-packages/sage/combinat/root_system/non_symmetric_macdonald_polynomials.py", line 1208, in __classcall__
        q = K(q)
      File "sage/structure/parent.pyx", line 1095, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9523)
        return mor._call_(x)
      File "sage/structure/coerce_maps.pyx", line 109, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4402)
        raise
      File "sage/structure/coerce_maps.pyx", line 104, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_ (build/cythonized/sage/structure/coerce_maps.c:4309)
        return C._element_constructor(x)
      File "/home/wluebbe/sage-6.8.beta3/local/lib/python2.7/site-packages/sage/rings/fraction_field.py", line 563, in _element_constructor_
        x = sage_eval(x, self.gens_dict_recursive())
      File "/home/wluebbe/sage-6.8.beta3/local/lib/python2.7/site-packages/sage/misc/sage_eval.py", line 181, in sage_eval
        if not isinstance(source, six.string_types):
    AttributeError: 'module' object has no attribute 'string_types'

Apparently the module six has not yet the attribute string_types. This seems to be the symptom of a circular import. But why after this last, small change?

[Wilfried Luebbe]

Apparently both fraction.field.py and sage-eval.py (see the last 2 listed files) contain import six. And at the point where the error occurs the import process for module six (started in fraction.field.py) is not yet complete.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7e60f88

prevent relative import of six

[aapitzsch]

There is a new six.py in misc which is relative imported instead of the expected six module.

[Wilfried Luebbe]

I did not think of a new six.py module in src/sage/misc!

All test passed. Looks good.

[Ralf Stephan]

You were not satisfied with my part of the review, sir?

[Wilfried Luebbe]

Replying to rws:

You were not satisfied with my part of the review, sir?

Sorry about that.