Sage: Ticket #22037: Upgrade to Python 2.7.13
https://trac.sagemath.org/ticket/22037
<p>
<strong>Tarball</strong>: <a class="extlink" href="https://www.python.org/ftp/python/2.7.13/Python2.7.13.tgz"><span class="icon"></span>https://www.python.org/ftp/python/2.7.13/Python2.7.13.tgz</a>
</p>
<p>
Note: upgrade to Python 2.7.12 done in <a class="closed ticket" href="https://trac.sagemath.org/ticket/19735" title="enhancement: Upgrade to Python 2.7.12 (closed: fixed)">#19735</a>.
</p>
enusSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/22037
Trac 1.1.6thansenWed, 07 Dec 2016 17:53:52 GMTdescription changed
https://trac.sagemath.org/ticket/22037#comment:1
https://trac.sagemath.org/ticket/22037#comment:1
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=1">diff</a>)
</li>
</ul>
TicketthansenWed, 07 Dec 2016 17:55:28 GMTdescription changed
https://trac.sagemath.org/ticket/22037#comment:2
https://trac.sagemath.org/ticket/22037#comment:2
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=2">diff</a>)
</li>
</ul>
TicketjdemeyerWed, 07 Dec 2016 18:49:02 GMTdescription changed
https://trac.sagemath.org/ticket/22037#comment:3
https://trac.sagemath.org/ticket/22037#comment:3
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=3">diff</a>)
</li>
</ul>
TicketthansenThu, 08 Dec 2016 23:22:59 GMTattachment set
https://trac.sagemath.org/ticket/22037
https://trac.sagemath.org/ticket/22037
<ul>
<li><strong>attachment</strong>
set to <em>example.tar.gz</em>
</li>
</ul>
<p>
This is a small example showing how the <a class="missing wiki">ClasscallMetaclass?</a> leads to the error.
</p>
TicketthansenFri, 09 Dec 2016 00:31:36 GMT
https://trac.sagemath.org/ticket/22037#comment:4
https://trac.sagemath.org/ticket/22037#comment:4
<p>
So we have
</p>
<p>
<code>class UniqueRepresentation(CachedRepresentation, WithEqualityById)</code>
</p>
<p>
where <code>CachedRepresentation</code> uses <code>ClasscallMetaclass</code> and <code>WithEqualityById</code> is a cdef class.
</p>
<p>
If I change that to
</p>
<p>
<code>class UniqueRepresentation(WithEqualityById, CachedRepresentation)</code>
</p>
<p>
the error goes away. I don't know if that would be a correct fix though.
</p>
TicketnthieryFri, 09 Dec 2016 22:07:52 GMT
https://trac.sagemath.org/ticket/22037#comment:5
https://trac.sagemath.org/ticket/22037#comment:5
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:4" title="Comment 4">thansen</a>:
</p>
<blockquote class="citation">
<p>
So we have
</p>
<p>
<code>class UniqueRepresentation(CachedRepresentation, WithEqualityById)</code>
</p>
<p>
where <code>CachedRepresentation</code> uses <code>ClasscallMetaclass</code> and <code>WithEqualityById</code> is a cdef class.
</p>
<p>
If I change that to
</p>
<p>
<code>class UniqueRepresentation(WithEqualityById, CachedRepresentation)</code>
</p>
<p>
the error goes away. I don't know if that would be a correct fix though.
</p>
</blockquote>
<p>
In principle, the features provided by the two super classes are
orthogonal (equality vs construction). Right now, I don't remember
enough the details to see whether there could be subtleties due to one
class being cdef. I'll dig further.
</p>
<p>
Simon, any opinion?
</p>
<p>
Cheers,
</p>
<blockquote>
<p>
Nicolas
</p>
</blockquote>
TicketnthieryFri, 09 Dec 2016 22:08:26 GMTcc set
https://trac.sagemath.org/ticket/22037#comment:6
https://trac.sagemath.org/ticket/22037#comment:6
<ul>
<li><strong>cc</strong>
<em>SimonKing</em> added
</li>
</ul>
TicketnthieryFri, 09 Dec 2016 22:11:09 GMTdescription changed
https://trac.sagemath.org/ticket/22037#comment:7
https://trac.sagemath.org/ticket/22037#comment:7
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=7">diff</a>)
</li>
</ul>
TicketthansenSat, 10 Dec 2016 13:34:39 GMT
https://trac.sagemath.org/ticket/22037#comment:8
https://trac.sagemath.org/ticket/22037#comment:8
<p>
Indeed Juliens example from <a class="extlink" href="https://bugs.python.org/issue25731"><span class="icon"></span>https://bugs.python.org/issue25731</a> also exposes this issue:
</p>
<pre class="wiki">$ cat foo.pxd
cdef class B:
cdef object b
$ cat foo.pyx
cdef class A:
pass
cdef class B:
def __init__(self, b):
self.b = b
$ cat bar.py
from foo import A, B
class C(A, B):
def __init__(self):
B.__init__(self, 1)
C()
$ cython foo.pyx && gcc I/usr/include/python2.7 Wall shared fPIC o foo.so foo.c
$ python c 'import bar'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "bar.py", line 7, in <module>
C()
TypeError: foo.A.__new__(C) is not safe, use foo.B.__new__()
</pre>
TicketnthierySat, 10 Dec 2016 22:15:32 GMT
https://trac.sagemath.org/ticket/22037#comment:9
https://trac.sagemath.org/ticket/22037#comment:9
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:4" title="Comment 4">thansen</a>:
</p>
<blockquote class="citation">
<p>
So we have
</p>
<p>
<code>class UniqueRepresentation(CachedRepresentation, WithEqualityById)</code>
</p>
<p>
where <code>CachedRepresentation</code> uses <code>ClasscallMetaclass</code> and <code>WithEqualityById</code> is a cdef class.
</p>
<p>
If I change that to
</p>
<p>
<code>class UniqueRepresentation(WithEqualityById, CachedRepresentation)</code>
</p>
<p>
the error goes away. I don't know if that would be a correct fix though.
</p>
</blockquote>
<p>
Looking back at the code, this still feels rather harmless.
</p>
<p>
I ran all long Sage tests; beside trivial doctests failures due to mro
printouts, the only errors all boil down to the following:
</p>
<pre class="wiki">sage t long src/sage/combinat/root_system/cartan_type.py
**********************************************************************
File "src/sage/combinat/root_system/cartan_type.py", line 2999, in sage.combinat.root_system.cartan_type.CartanType_simple_finite.__setstate__
Failed example:
unpickle_build(si1, {'tools':pg_unpickleModule('sage.combinat.root_system.type_A'), 't':['A', si2], 'letter':'A', 'n':si2})
Exception raised:
Traceback (most recent call last):
File "/opt/sagegit2/local/lib/python2.7/sitepackages/sage/combinat/root_system/cartan_type.py", line 3014, in __setstate__
self.__class__ = T.__class__
TypeError: __class__ assignment: 'CartanType_simple_finite' object layout differs from 'CartanType'
</pre><p>
This is in a stub class whose only purpose is to support unpickling
rather old pickles (Sage <= 4.0). It's a stub for an old class that
does not exist anymore; upon unpickling an object, the <code>set_state</code>
method substitutes its stub class by the appropriate new class. This
is unsurprisingly fragile. I am unsure why it did not fail before and
fails now.
</p>
<p>
In any cases, making the stub class derive from <code>UniqueRepresentation</code>
ensures that the two classes have the same layout, which is slightly
cleaner than it was earlier. So that sounds like an acceptable fix.
</p>
<p>
I am about to push a branch with this fix and trivially updated
doctests where needed. Let's see if the patchbot confirms that all
test pass.
</p>
<p>
Still would be happy to have feedback from Simon.
</p>
<p>
Cheers,
</p>
<blockquote>
<p>
Nicolas
</p>
</blockquote>
TicketnthierySat, 10 Dec 2016 22:24:10 GMTbranch set
https://trac.sagemath.org/ticket/22037#comment:10
https://trac.sagemath.org/ticket/22037#comment:10
<ul>
<li><strong>branch</strong>
set to <em>u/nthiery/upgrade_to_python_2_7_13</em>
</li>
</ul>
TicketnthierySat, 10 Dec 2016 22:27:11 GMTcommit set
https://trac.sagemath.org/ticket/22037#comment:11
https://trac.sagemath.org/ticket/22037#comment:11
<ul>
<li><strong>commit</strong>
set to <em>b7bd4597e18af490dd9e6225e47b16d5ec8d79dc</em>
</li>
</ul>
<p>
Note: not being super familiar with the process, the above branch does *not* do the upgrade to Python 2.7.13; just the workaround suggested by Tobbias.
</p>
<hr />
<p>
New commits:
</p>
<table class="wiki">
<tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=8968314bec3ce7545da7465274494d9dd5be0dc3"><span class="icon"></span>8968314</a></td><td><code>22037: switch order of the bases of the class UniqueRepresentation</code>
</td></tr><tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=b7bd4597e18af490dd9e6225e47b16d5ec8d79dc"><span class="icon"></span>b7bd459</a></td><td><code>22037: more robust backwardcompatibility stub class CartanType_simple_finite</code>
</td></tr></table>
TicketthansenSat, 10 Dec 2016 22:33:18 GMT
https://trac.sagemath.org/ticket/22037#comment:12
https://trac.sagemath.org/ticket/22037#comment:12
<p>
There are several places that need to be changed. The problem is that the errors are discovered one by one. Is there a standard way to rebuild just a single .pyx file? Right now I have to do a rebuild of sagelib everytime I change one. This is my patch right now, and there are still new errors coming up:
</p>
<pre class="wiki"> a/src/sage/structure/unique_representation.py
+++ b/src/sage/structure/unique_representation.py
@@ 1176,7 +1176,7 @@
return cls(*args, **keywords)
class UniqueRepresentation(CachedRepresentation, WithEqualityById):
+class UniqueRepresentation(WithEqualityById, CachedRepresentation):
r"""
Classes derived from UniqueRepresentation inherit a unique
representation behavior for their instances.
 a/src/sage/categories/cartesian_product.py
+++ b/src/sage/categories/cartesian_product.py
@@ 18,7 +18,7 @@
native_python_containers = set([tuple, list, set, frozenset])
class CartesianProductFunctor(CovariantFunctorialConstruction, MultivariateConstructionFunctor):
+class CartesianProductFunctor(MultivariateConstructionFunctor, CovariantFunctorialConstruction):
"""
A singleton class for the Cartesian product functor.
 a/src/sage/rings/infinity.py
+++ b/src/sage/rings/infinity.py
@@ 544,7 +544,7 @@
class UnsignedInfinityRing_class(Singleton, Ring):

+ __new__ = Ring.__new__
def __init__(self):
"""
Initialize ``self``.
@@ 950,6 +950,7 @@
pass
class InfinityRing_class(Singleton, Ring):
+ __new__ = Ring.__new__
def __init__(self):
"""
Initialize ``self``.
 a/src/sage/structure/formal_sum.py
+++ b/src/sage/structure/formal_sum.py
@@ 314,7 +314,7 @@
w.append((coeff,last))
self._data = w
class FormalSums(UniqueRepresentation, Module):
+class FormalSums(Module, UniqueRepresentation):
"""
The Rmodule of finite formal sums with coefficients in some ring R.
 a/src/sage/misc/inherit_comparison.pyx
+++ b/src/sage/misc/inherit_comparison.pyx
@@ 83,5 +83,5 @@
t.tp_compare = b.tp_compare
super(InheritComparisonMetaclass, self).__init__(*args)
class InheritComparisonClasscallMetaclass(InheritComparisonMetaclass, ClasscallMetaclass):
+class InheritComparisonClasscallMetaclass(ClasscallMetaclass, InheritComparisonMetaclass):
pass
 a/src/sage/rings/polynomial/polynomial_ring.py
+++ b/src/sage/rings/polynomial/polynomial_ring.py
@@ 1439,7 +1439,7 @@
return self._monics_max( max_degree )
raise ValueError("you should pass exactly one of of_degree and max_degree")
class PolynomialRing_commutative(PolynomialRing_general, commutative_algebra.CommutativeAlgebra):
+class PolynomialRing_commutative(commutative_algebra.CommutativeAlgebra, PolynomialRing_general):
"""
Univariate polynomial ring over a commutative ring.
"""
 a/src/sage/rings/real_arb.pyx
+++ b/src/sage/rings/real_arb.pyx
@@ 355,7 +355,7 @@
if _do_sig(myprec): sig_off()
return 0
class RealBallField(UniqueRepresentation, Field):
+class RealBallField(Field, UniqueRepresentation):
r"""
An approximation of the field of real numbers using midrad intervals, also
known as balls.
</pre>
TicketnthierySat, 10 Dec 2016 22:39:24 GMT
https://trac.sagemath.org/ticket/22037#comment:13
https://trac.sagemath.org/ticket/22037#comment:13
<p>
Ouch, this is getting more annoying. Let's see how big the list grows. It's more worrying to have to move <a class="missing wiki">UniqueRepresentation?</a> to second place.
</p>
<p>
<code>sage b</code> rebuilds just the Sage library. This is usually good enough when only a single pyx file needs to be rebuilt.
</p>
<p>
Have a good night,
</p>
TicketthansenSat, 10 Dec 2016 22:45:40 GMT
https://trac.sagemath.org/ticket/22037#comment:14
https://trac.sagemath.org/ticket/22037#comment:14
<p>
Is better not to change the order of base classes and instead just define <code>__new__</code> everywhere like I did for example for InfinityRing_class?
</p>
TicketnthierySun, 11 Dec 2016 07:56:52 GMT
https://trac.sagemath.org/ticket/22037#comment:15
https://trac.sagemath.org/ticket/22037#comment:15
<p>
Ah, I had missed that. If this works, this would be acceptable as a temporary workaround.
My first guess is that this would break the <a class="missing wiki">CachedRepresentation?</a>; but I'll give it a try.
</p>
<p>
Anyone available for making a branch with Python 2.7.13, so that we can all easilly experiment with it? Otherwise, I'll try to squeeze this sometime today, but really no promise.
</p>
<p>
Thanks Tobias for investigating this hard!
</p>
TicketnthierySun, 11 Dec 2016 07:57:16 GMTcc changed
https://trac.sagemath.org/ticket/22037#comment:16
https://trac.sagemath.org/ticket/22037#comment:16
<ul>
<li><strong>cc</strong>
<em>hivert</em> added
</li>
</ul>
TicketslelievreSun, 11 Dec 2016 09:41:06 GMTcc, description changed
https://trac.sagemath.org/ticket/22037#comment:17
https://trac.sagemath.org/ticket/22037#comment:17
<ul>
<li><strong>cc</strong>
<em>slelievre</em> added
</li>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=17">diff</a>)
</li>
</ul>
<p>
<a class="closed ticket" href="https://trac.sagemath.org/ticket/19735" title="enhancement: Upgrade to Python 2.7.12 (closed: fixed)">#19735</a> "Upgrade to Python 2.7.12" was just closed.
</p>
TicketjdemeyerSun, 11 Dec 2016 12:55:57 GMT
https://trac.sagemath.org/ticket/22037#comment:18
https://trac.sagemath.org/ticket/22037#comment:18
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:13" title="Comment 13">nthiery</a>:
</p>
<blockquote class="citation">
<p>
<code>sage b</code> rebuilds just the Sage library. This is usually good enough when only a single pyx file needs to be rebuilt.
</p>
</blockquote>
<p>
Although it should be said that <code>sage b</code> should be used only if you "know what you are doing". It's safer to run <code>make sagelib</code> (build Sage library) or <code>make build</code> (build all Sage packages) or plain <code>make</code> (build Sage + documentation). The <code>make</code> commands have dependency checking, so an outdated library will be rebuilt if needed. When you only run <code>sage b</code>, you might get into trouble when switching branches because some dependencies might not be updated.
</p>
TicketthansenSun, 11 Dec 2016 13:04:41 GMT
https://trac.sagemath.org/ticket/22037#comment:19
https://trac.sagemath.org/ticket/22037#comment:19
<p>
Thanks. I'm using Debian packages for the dependencies so I'm just rebuilding sagelib anyway. I was asking if it was possible to rebuild just a single .pyx file.
</p>
<p>
Meanwhile the patch is getting bigger and bigger. I think I need to add <code>__new__ = Parent.__new__</code> to all classes that have the base classes "Uniquerepresentation, Parent", which are many. And others. At least now sage starts and one can run the tests to uncover more errors. Should I keep going on with this?
</p>
<pre class="wiki"> a/src/sage/structure/unique_representation.py
+++ b/src/sage/structure/unique_representation.py
@@ 1342,3 +1342,4 @@
....:
sage: b = bla()
"""
+ __new__ = object.__new__
 a/src/sage/categories/cartesian_product.py
+++ b/src/sage/categories/cartesian_product.py
@@ 105,6 +105,7 @@
:class:`CartesianProductsCategory`.
"""
+ __new__ = MultivariateConstructionFunctor.__new__
_functor_name = "cartesian_product"
_functor_category = "CartesianProducts"
symbol = " (+) "
 a/src/sage/rings/infinity.py
+++ b/src/sage/rings/infinity.py
@@ 544,7 +544,7 @@
class UnsignedInfinityRing_class(Singleton, Ring):

+ __new__ = Ring.__new__
def __init__(self):
"""
Initialize ``self``.
@@ 950,6 +950,7 @@
pass
class InfinityRing_class(Singleton, Ring):
+ __new__ = Ring.__new__
def __init__(self):
"""
Initialize ``self``.
 a/src/sage/structure/formal_sum.py
+++ b/src/sage/structure/formal_sum.py
@@ 336,6 +336,7 @@
sage: TestSuite(FormalSums(QQ)).run()
"""
+ __new__ = Module.__new__
Element = FormalSum
@staticmethod
def __classcall__(cls, base_ring = ZZ):
 a/src/sage/misc/inherit_comparison.pyx
+++ b/src/sage/misc/inherit_comparison.pyx
@@ 84,4 +84,5 @@
super(InheritComparisonMetaclass, self).__init__(*args)
class InheritComparisonClasscallMetaclass(InheritComparisonMetaclass, ClasscallMetaclass):
+ __new__ = ClasscallMetaclass.__new__
pass
 a/src/sage/rings/polynomial/polynomial_ring.py
+++ b/src/sage/rings/polynomial/polynomial_ring.py
@@ 1443,6 +1443,7 @@
"""
Univariate polynomial ring over a commutative ring.
"""
+ __new__ = commutative_algebra.CommutativeAlgebra.__new__
def __init__(self, base_ring, name=None, sparse=False, element_class=None, category=None):
if base_ring not in _CommutativeRings:
raise TypeError("Base ring %s must be a commutative ring."%repr(base_ring))
 a/src/sage/rings/real_arb.pyx
+++ b/src/sage/rings/real_arb.pyx
@@ 398,6 +398,7 @@
sage: RBF.zero()
0
"""
+ __new__ = Field.__new__
Element = RealBall
@staticmethod
 a/src/sage/structure/dynamic_class.py
+++ b/src/sage/structure/dynamic_class.py
@@ 460,6 +460,7 @@
pass
class DynamicInheritComparisonMetaclass(DynamicMetaclass, InheritComparisonMetaclass):
+ __new__ = type.__new__
pass
class DynamicInheritComparisonClasscallMetaclass(DynamicMetaclass, InheritComparisonClasscallMetaclass):
 a/src/sage/rings/complex_arb.pyx
+++ b/src/sage/rings/complex_arb.pyx
@@ 234,6 +234,7 @@
...
ValueError: Precision must be at least 2.
"""
+ __new__ = Field.__new__
Element = ComplexBall
@staticmethod
 a/src/sage/rings/pari_ring.py
+++ b/src/sage/rings/pari_ring.py
@@ 165,6 +165,7 @@
sage: loads(R.dumps()) is R
True
"""
+ __new__ = ring.Ring.__new__
Element = Pari
def __init__(self):
 a/src/sage/rings/contfrac.py
+++ b/src/sage/rings/contfrac.py
@@ 78,6 +78,7 @@
sage: CFF.category()
Category of fields
"""
+ __new__ = Field.__new__
class Element(ContinuedFraction_periodic,FieldElement):
r"""
A continued fraction of a rational number.
 a/src/sage/rings/number_field/number_field.py
+++ b/src/sage/rings/number_field/number_field.py
@@ 1264,6 +1264,7 @@
....: assert elt.is_integral()
"""
+ __new__ = number_field_base.NumberField.__new__
def __init__(self, polynomial, name, latex_name,
check=True, embedding=None, category=None,
assume_disc_small=False, maximize_at_primes=None, structure=None):
 a/src/sage/matrix/matrix_space.py
+++ b/src/sage/matrix/matrix_space.py
@@ 137,6 +137,7 @@
...
ValueError: number of rows and columns may be at most...
"""
+ __new__ = parent_gens.ParentWithGens.__new__
_no_generic_basering_coercion = True
@staticmethod
 a/src/sage/schemes/generic/scheme.py
+++ b/src/sage/schemes/generic/scheme.py
@@ 791,6 +791,7 @@
:class:`sage.schemes.generic.algebraic_scheme.AffineSpace`.
"""
+ __new__ = Scheme.__new__
def __init__(self, R, S=None, category=None):
"""
Construct the affine scheme with coordinate ring `R`.
 a/src/sage/combinat/partition.py
+++ b/src/sage/combinat/partition.py
@@ 5061,6 +5061,7 @@
sage: P.list()
[[3, 2]]
"""
+ __new__ = Parent.__new__
@staticmethod
def __classcall_private__(cls, n=None, **kwargs):
"""
 a/src/sage/sets/non_negative_integers.py
+++ b/src/sage/sets/non_negative_integers.py
@@ 67,7 +67,7 @@
TODO: do not use ``NN`` any more in the doctests for
``NonNegativeIntegers``.
"""

+ __new__ = Parent.__new__
def __init__(self, category=None):
"""
TESTS::
 a/src/sage/structure/sequence.py
+++ b/src/sage/structure/sequence.py
@@ 410,6 +410,7 @@
Finite Field of size 5
"""
+ __new__ = list.__new__
def __init__(self, x, universe=None, check=True, immutable=False,
cr=False, cr_str=None, use_sage_types=False):
"""
 a/src/sage/structure/list_clone_demo.pyx
+++ b/src/sage/structure/list_clone_demo.pyx
@@ 61,7 +61,7 @@
sage: IncreasingArrays().element_class
<type 'sage.structure.list_clone_demo.IncreasingArray'>
"""

+ __new__ = Parent.__new__
def __init__(self):
"""
TESTS::
 a/src/sage/sets/finite_enumerated_set.py
+++ b/src/sage/sets/finite_enumerated_set.py
@@ 80,7 +80,7 @@
sage: S.first().parent()
Integer Ring
"""

+ __new__ = Parent.__new__
@staticmethod
def __classcall__(cls, iterable):
"""
 a/src/sage/combinat/derangements.py
+++ b/src/sage/combinat/derangements.py
@@ 130,6 +130,7 @@
sage: D2.random_element() # random
[2, 3, 1, 3, 1, 2]
"""
+ __new__ = Parent.__new__
@staticmethod
def __classcall_private__(cls, x):
"""
 a/src/sage/combinat/permutation.py
+++ b/src/sage/combinat/permutation.py
@@ 5003,6 +5003,7 @@
sage: p.random_element()
[5, 1, 2, 4, 3]
"""
+ __new__ = Parent.__new__
@staticmethod
def __classcall_private__(cls, n=None, k=None, **kwargs):
"""
 a/src/sage/categories/examples/infinite_enumerated_sets.py
+++ b/src/sage/categories/examples/infinite_enumerated_sets.py
@@ 73,7 +73,7 @@
running ._test_pickling() . . . pass
running ._test_some_elements() . . . pass
"""

+ __new__ = Parent.__new__
def __init__(self):
"""
TESTS::
 a/src/sage/categories/examples/sets_with_grading.py
+++ b/src/sage/categories/examples/sets_with_grading.py
@@ 23,6 +23,7 @@
sage: E.graded_component(100)
{100}
"""
+ __new__ = Parent.__new__
def __init__(self):
r"""
TESTS::
 a/src/sage/sets/cartesian_product.py
+++ b/src/sage/sets/cartesian_product.py
@@ 53,6 +53,7 @@
.. automethod:: _cartesian_product_of_elements
"""
+ __new__ = Parent.__new__
def __init__(self, sets, category, flatten=False):
r"""
INPUT:
</pre>
TicketjdemeyerSun, 11 Dec 2016 13:14:40 GMTdescription changed
https://trac.sagemath.org/ticket/22037#comment:20
https://trac.sagemath.org/ticket/22037#comment:20
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=20">diff</a>)
</li>
</ul>
TicketjdemeyerSun, 11 Dec 2016 13:32:31 GMTbranch changed
https://trac.sagemath.org/ticket/22037#comment:21
https://trac.sagemath.org/ticket/22037#comment:21
<ul>
<li><strong>branch</strong>
changed from <em>u/nthiery/upgrade_to_python_2_7_13</em> to <em>u/jdemeyer/upgrade_to_python_2_7_13</em>
</li>
</ul>
TicketjdemeyerSun, 11 Dec 2016 13:34:38 GMTcommit changed
https://trac.sagemath.org/ticket/22037#comment:22
https://trac.sagemath.org/ticket/22037#comment:22
<ul>
<li><strong>commit</strong>
changed from <em>b7bd4597e18af490dd9e6225e47b16d5ec8d79dc</em> to <em>ba486e12515ad417e2aa747dc5e525e0bedd3d4d</em>
</li>
</ul>
<p>
This branch now contains both the upgrade to Python 2.7.13rc1 as well as the commits by Nicolas.
</p>
<hr />
<p>
New commits:
</p>
<table class="wiki">
<tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=c35ea3cbcef6c1ae9f6a939af853ac339fcd691c"><span class="icon"></span>c35ea3c</a></td><td><code>Upgrade python2 to 2.7.12 and update tinfo and uuid patches</code>
</td></tr><tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=85e931baf9d76af3b4f011d6b499227b6dd4aea7"><span class="icon"></span>85e931b</a></td><td><code>fix sage_timeit and word.py for python 2.7.12</code>
</td></tr><tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=a119bf7eb0597ef25016b070f5eaedbb280f4ed4"><span class="icon"></span>a119bf7</a></td><td><code>Upgrade to Python 2.7.13rc1</code>
</td></tr><tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=a735fa5dc3ea81bc316f8a9cdcb013160382f770"><span class="icon"></span>a735fa5</a></td><td><code>22037: switch order of the bases of the class UniqueRepresentation</code>
</td></tr><tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit?id=ba486e12515ad417e2aa747dc5e525e0bedd3d4d"><span class="icon"></span>ba486e1</a></td><td><code>22037: more robust backwardcompatibility stub class CartanType_simple_finite</code>
</td></tr></table>
TicketthansenSun, 11 Dec 2016 17:54:18 GMT
https://trac.sagemath.org/ticket/22037#comment:23
https://trac.sagemath.org/ticket/22037#comment:23
<p>
I now have a patch that changes more than 100 files and that's still not all. I don't think fixing this casebycase in this way is viable. In any case this would be just a temporary fix. If we could find a simpler workaround in Sage that would be great, otherwise we'll have to try to get the Python change reverted I guess.
</p>
TicketjdemeyerSun, 11 Dec 2016 18:45:50 GMT
https://trac.sagemath.org/ticket/22037#comment:24
https://trac.sagemath.org/ticket/22037#comment:24
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:23" title="Comment 23">thansen</a>:
</p>
<blockquote class="citation">
<p>
we'll have to try to get the Python change reverted I guess.
</p>
</blockquote>
<p>
I have no idea how feasible that is, but that would be the best solution. There is clearly a backwardsincompatible change being done, so that would be a good argument to revert the change.
</p>
TicketnthierySun, 11 Dec 2016 18:47:19 GMT
https://trac.sagemath.org/ticket/22037#comment:25
https://trac.sagemath.org/ticket/22037#comment:25
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:23" title="Comment 23">thansen</a>:
</p>
<blockquote class="citation">
<p>
I now have a patch that changes more than 100 files and that's still not all. I don't think fixing this casebycase in this way is viable.
</p>
</blockquote>
<p>
Sounds like it indeed.
</p>
<blockquote class="citation">
<p>
In any case this would be just a temporary fix. If we could find a simpler workaround in Sage that would be great, otherwise we'll have to try to get the Python change reverted I guess.
</p>
</blockquote>
<p>
I assume you have tried to do this <code>xxx.__new__</code> change in the <code>UniqueRepresentation</code> base class, and that did not work, right? Plausibly because which <code>xxx</code> shall be used depends from one case to the other. Maybe the <code>ClassCallMetaclass.__init__</code> method could handle this upon class creation, figuring out the proper <code>xxx</code> to use.
</p>
<p>
I am currently recompiling Sage with the new branch (thanks Jeroen), and will try to look at this later tonight.
</p>
<p>
I tchated with Florent earlier today; he was sitting next to Julien last Spring when they investigated this. He will have a look as well and try to remember and post the details.
</p>
TicketnthierySun, 11 Dec 2016 18:48:36 GMT
https://trac.sagemath.org/ticket/22037#comment:26
https://trac.sagemath.org/ticket/22037#comment:26
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:24" title="Comment 24">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:23" title="Comment 23">thansen</a>:
</p>
<blockquote class="citation">
<p>
we'll have to try to get the Python change reverted I guess.
</p>
</blockquote>
<p>
I have no idea how feasible that is, but that would be the best solution. There is clearly a backwardsincompatible change being done, so that would be a good argument to revert the change.
</p>
</blockquote>
<p>
+1
</p>
<p>
(revert or possibly ammend the change so that it would not break the backward compatibility, but still achieve the desired feature they wanted)
</p>
TicketthansenSun, 11 Dec 2016 18:56:53 GMT
https://trac.sagemath.org/ticket/22037#comment:27
https://trac.sagemath.org/ticket/22037#comment:27
<p>
If we can revert this in the Python package in Debian in time depends on what the maintainer thinks and if he responds quickly.
</p>
<p>
It would help if the change would break another software that is already in Debian. In that case the bug would have a higher severity (RC bug) and that would force the maintainer to do something or allow us to upload a change. I already rebuilt some packages to see if they are affected (numpy, scipy, etc) but they all seem to be fine. If someone knows software that uses inheritance with cdef classes and is in Debian let me know.
</p>
TicketjdemeyerSun, 11 Dec 2016 19:59:39 GMT
https://trac.sagemath.org/ticket/22037#comment:28
https://trac.sagemath.org/ticket/22037#comment:28
<p>
Here is a more minimal breaking example:
</p>
<p>
<strong>obj.pyx</strong>:
</p>
<pre class="wiki">cdef class OBJ(object):
pass
</pre><p>
Compile this (within Sage) with
</p>
<pre class="wiki">./sage sh c "cython obj.pyx && gcc Ilocal/include/python2.7 Wall shared fPIC obj.c o obj.so"
</pre><p>
With Python 2.7.13rc1, this gives
</p>
<pre class="wiki">In [1]: from obj import OBJ
In [2]: class X(OBJ, dict):
...: pass
...:
In [3]: X()

TypeError Traceback (most recent call last)
<ipythoninput3a7d4f7b89654> in <module>()
> 1 X()
TypeError: obj.OBJ.__new__(X) is not safe, use dict.__new__()
</pre>
TickethivertSun, 11 Dec 2016 22:55:10 GMT
https://trac.sagemath.org/ticket/22037#comment:29
https://trac.sagemath.org/ticket/22037#comment:29
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:28" title="Comment 28">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
Here is a more minimal breaking example:
</p>
</blockquote>
<p>
Hi there,
</p>
<p>
Here is what I understand:
</p>
<ul><li>When creating a class, Python has to decide which function needs to be called
to allocate (<code>__new__</code> method) an object. As the Clevel, the function is
put in a particular field called tp_new of the Cstruct corresponding to the
class.
</li><li>In case of multiple inheritance, this is only possible if the two classes
has a compatible memory layout. In this case, if one of the class has a
specific extra Cattribute, its <code>__new__</code> must be called (if both classes have
extra attributes then they are not layout compatible).
</li><li>Before the change of <a class="extlink" href="https://bugs.python.org/issue25731"><span class="icon"></span>https://bugs.python.org/issue25731</a>, CPython had a slow
way to determine which must be called (see comment at
<a class="extlink" href="https://hg.python.org/cpython/rev/e7062dd9085e/"><span class="icon"></span>https://hg.python.org/cpython/rev/e7062dd9085e/</a>). When trying to optimize
that, they introduced a bug, which Julien investigate with my help. As a
consequence the change was reverted.
</li><li>In <a class="extlink" href="https://hg.python.org/cpython/rev/a37cc3d926ec"><span class="icon"></span>https://hg.python.org/cpython/rev/a37cc3d926ec</a>, they reintroduced the
fast method issuing a warning if the wrong <code>__new__</code> is called. As far as I
understand, this means that in case of multiple inheritance with Cython
code, only the first class can extend the Cstruct.
</li></ul><p>
Note that I have to investigate further to be sure of this last point.
</p>
TicketnthieryMon, 12 Dec 2016 22:33:39 GMT
https://trac.sagemath.org/ticket/22037#comment:30
https://trac.sagemath.org/ticket/22037#comment:30
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:25" title="Comment 25">nthiery</a>:
</p>
<blockquote class="citation">
<p>
I assume you have tried to do this <code>xxx.__new__</code> change in the
<code>UniqueRepresentation</code> base class, and that did not work, right?
Plausibly because which <code>xxx</code> shall be used depends from one case to
the other. Maybe the <code>ClassCallMetaclass.__init__</code> method could
handle this upon class creation, figuring out the proper <code>xxx</code> to
use.
</p>
</blockquote>
<p>
I investigated this a bit further. First I believe we don't override
<code>object.__new__</code> in any of our Cython classes, and in only very few of
our Python classes:
</p>
<pre class="wiki">mistral/opt/sage/src/sage>grep "def __new__" **/*.py*
combinat/shard_order.py: def __new__(cls, p):
combinat/words/alphabet.py: def __new__(self, alphabet=None, name=None):
combinat/words/finite_word.py: def __new__(cls, words):
misc/classcall_metaclass.pyx: def __new__(*args):
misc/classcall_metaclass.pyx: ....: def __new__(cls):
misc/sage_input.py: def __new__(cls, cmds, expr, locals=None):
misc/six.py: def __new__(cls, name, _, __dict__):
rings/infinity.py: def __new__(cls, *args):
rings/qqbar.py: def __new__(cls):
rings/qqbar.py: def __new__(cls):
rings/qqbar.py: def __new__(self, generator, value):
rings/qqbar.py: def __new__(self, a, b):
rings/rational_field.py: def __new__(cls):
</pre><p>
Therefore, resetting <code>cls.__new__</code> to be just <code>object.__new__</code> for all
our classes in the <code>ClassCallMetaclass</code> should be safe. So I added
just that at the end of <code>ClassCallMetaclass.__init__</code>:
</p>
<pre class="wiki"> self.__new__ = object.__new__
</pre><p>
This seems to do the job for e.g. <code>Unknown</code> and a couple others.
</p>
<p>
However this does not seem to be quite what we want: it sounds like
<code>object.__new__</code> is "bound" to the class object, and I believe that's
what forced Tobias to use e.g. <code>__new__ = Functor.__new__</code> in his
patch. Instead, what we would want to do is the analog of plain
Python's
</p>
<pre class="wiki"> self.__new__ = object.__new__.im_func
</pre><p>
to make sure that <code>self.__new__</code> is bound to <code>self</code> and not to
<code>object</code>. I have tried to play around with what we do in a similar
context in <code>sage.structure.misc.getattr_from_other_class</code>, but did not
quite manage to. Trying to fetch <code>object.__new__.__get__</code> with
<code>Py_TYPE(attribute).tp_descr_get</code> returns null.
</p>
<p>
Florent, Jeroen, or anyone with good knowledge of the CPython API,
could you have a look? At the end of the day, this is all probably
emulating the proper initialization of the <code>tp_new</code> attribute that's
been broken by Python's change. Maybe it's in fact just enough to
emulate this initialization for the <code>WithEqualityById</code> class. Or just
define a trivial <code>WithEqualityById.__new__</code>?
</p>
<p>
Florent: will you be at LRI tomorrow? If yes, we can have a sprint
together on this.
</p>
<p>
While I am fairly optimistic on having a workaround soon, it still
would be good to investigate in parallel getting this fixed / patched
in Python.
</p>
<p>
Tobias: to answer your question, on my side, I haven't heard of other
soft using inheritance from multiple Cython classes.
</p>
<p>
Cheers,
</p>
<blockquote>
<p>
Nicolas
</p>
</blockquote>
TicketjdemeyerTue, 13 Dec 2016 08:27:36 GMT
https://trac.sagemath.org/ticket/22037#comment:31
https://trac.sagemath.org/ticket/22037#comment:31
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:30" title="Comment 30">nthiery</a>:
</p>
<blockquote class="citation">
<p>
First I believe we don't override
<code>object.__new__</code> in any of our Cython classes
</p>
</blockquote>
<p>
Not in the literal sense. However, Cython always implements a custom <code>tp_new</code> (which is the C version of <code>__new__</code>) for any <code>cdef class</code>. So effectively, we are overriding <code>__new__</code> in every <code>cdef class</code>.
</p>
<blockquote class="citation">
<p>
Therefore, resetting <code>cls.__new__</code> to be just <code>object.__new__</code> for all
our classes in the <code>ClassCallMetaclass</code> should be safe.
</p>
</blockquote>
<p>
Because of the above, I am afraid this is not true.
</p>
TicketjdemeyerTue, 13 Dec 2016 08:30:12 GMT
https://trac.sagemath.org/ticket/22037#comment:32
https://trac.sagemath.org/ticket/22037#comment:32
<p>
As I mentioned in <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a>, I think that breaking multiple inheritance of <code>__new__</code> is a sideeffect of the fix and not a fundamental feature needed to fix that bug. So there might be some hope of fixing <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a> without breaking multiple inheritance of <code>__new__</code>.
</p>
TicketnthieryWed, 14 Dec 2016 22:23:08 GMT
https://trac.sagemath.org/ticket/22037#comment:33
https://trac.sagemath.org/ticket/22037#comment:33
<p>
Florent: for whatever it's worth (not much) I pushed some changes I experimented with using <code>ClassCallMetaclass.__init__</code>:
</p>
<p>
<a class="extlink" href="https://git.sagemath.org/sage.git/log/?h=u/nthiery/22037/upgrade_to_python_2_7_13_classcall_workaround"><span class="icon"></span>https://git.sagemath.org/sage.git/log/?h=u/nthiery/22037/upgrade_to_python_2_7_13_classcall_workaround</a>
</p>
TicketnthieryWed, 14 Dec 2016 22:29:02 GMT
https://trac.sagemath.org/ticket/22037#comment:34
https://trac.sagemath.org/ticket/22037#comment:34
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:32" title="Comment 32">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
As I mentioned in <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a>, I think that breaking multiple inheritance of <code>__new__</code> is a sideeffect of the fix and not a fundamental feature needed to fix that bug. So there might be some hope of fixing <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a> without breaking multiple inheritance of <code>__new__</code>.
</p>
</blockquote>
<p>
That would be the ideal solution indeed!
</p>
<p>
Am I interpreting it correctly that the hunk causing trouble for us as just been reverted?
</p>
<p>
<a class="extlink" href="https://bugs.python.org/issue5322#msg283170"><span class="icon"></span>https://bugs.python.org/issue5322#msg283170</a>
</p>
<p>
If yes, does that resolve our issue for the Debian packaging?
</p>
TicketnthieryWed, 14 Dec 2016 22:30:59 GMT
https://trac.sagemath.org/ticket/22037#comment:35
https://trac.sagemath.org/ticket/22037#comment:35
<p>
Otherwise, worst comes to worst, we could consider the following tentative workaround: making <code>WithEqualityById</code> into a plain Cython class. This should be easy and only result in a speed regression, presumably not critical.
</p>
<p>
Cheers,
</p>
TicketthansenWed, 14 Dec 2016 22:35:19 GMT
https://trac.sagemath.org/ticket/22037#comment:36
https://trac.sagemath.org/ticket/22037#comment:36
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:34" title="Comment 34">nthiery</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:32" title="Comment 32">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
As I mentioned in <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a>, I think that breaking multiple inheritance of <code>__new__</code> is a sideeffect of the fix and not a fundamental feature needed to fix that bug. So there might be some hope of fixing <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a> without breaking multiple inheritance of <code>__new__</code>.
</p>
</blockquote>
<p>
That would be the ideal solution indeed!
</p>
<p>
Am I interpreting it correctly that the hunk causing trouble for us as just been reverted?
</p>
<p>
<a class="extlink" href="https://bugs.python.org/issue5322#msg283170"><span class="icon"></span>https://bugs.python.org/issue5322#msg283170</a>
</p>
<p>
If yes, does that resolve our issue for the Debian packaging?
</p>
</blockquote>
<p>
Yes it has been reverted upstream. It probably solves the problem. The maintainer of the Python package said he hopes a new Python version will be released this weekend and in this case he will upload it quickly to Debian. If no new Python is released we don't have a statement from him but I hope he would apply a patch to revert the change then.
</p>
TicketnthieryWed, 14 Dec 2016 22:43:33 GMT
https://trac.sagemath.org/ticket/22037#comment:37
https://trac.sagemath.org/ticket/22037#comment:37
<p>
Nice! Crossing my fingers ...
</p>
TicketnthieryWed, 14 Dec 2016 22:44:00 GMT
https://trac.sagemath.org/ticket/22037#comment:38
https://trac.sagemath.org/ticket/22037#comment:38
<p>
Just for the record: in the meantime, I had made a quick experiment with <code>WithEqualityById</code> as plain Python class, and got confused, as I still get the same error as before:
</p>
<pre class="wiki"> Unknown = UnknownClass()
...
TypeError: sage.structure.sage_object.SageObject.__new__(UnknownClass) is not safe, use object.__new__()
</pre><p>
whereas <code>UnknowClass</code> now has only one Cython base class; here is its mro:
</p>
<pre class="wiki">(<class 'sage.misc.unknown.UnknownClass'>, <class 'sage.structure.unique_representation.UniqueRepresentation'>, <class 'sage.structure.unique_representation.CachedRepresentation'>, <class 'sage.misc.fast_methods.WithEqualityById'>, <type 'sage.structure.sage_object.SageObject'>, <type 'object'>)
</pre>
TicketthansenMon, 19 Dec 2016 11:57:17 GMT
https://trac.sagemath.org/ticket/22037#comment:39
https://trac.sagemath.org/ticket/22037#comment:39
<p>
Python 2.7.13 was released without this change that breaks Sage, and already uploaded to Debian. So this is no longer an issue, unless Python really applies this change in a major version update one day. Thanks everybody for helping on this unexpected last minute obstruction!
</p>
<p>
BTW, Sage just hit Debian experimental today and we will upload it to unstable probably Wednesday or so. Cross your fingers!
</p>
TicketcharpentTue, 03 Jan 2017 22:38:21 GMTcc changed
https://trac.sagemath.org/ticket/22037#comment:40
https://trac.sagemath.org/ticket/22037#comment:40
<ul>
<li><strong>cc</strong>
<em>charpent</em> added
</li>
</ul>
<p>
I'd like to point people following this ticket to <a class="extlink" href="https://groups.google.com/forum/#!topic/sagedevel/jdLfIKQ1M18"><span class="icon"></span>this thread</a> which explains that the present ticket has some urgency. I attempted a workaround in <a class="closed ticket" href="https://trac.sagemath.org/ticket/22089" title="defect: Patch python to accept openSSL >= 1.1 (closed: wontfix)">#22089</a> and failed miserably.
</p>
<p>
Could you let us know what remains to be done to have this ticket completed ?
</p>
TicketjdemeyerWed, 04 Jan 2017 08:04:51 GMT
https://trac.sagemath.org/ticket/22037#comment:41
https://trac.sagemath.org/ticket/22037#comment:41
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:40" title="Comment 40">charpent</a>:
</p>
<blockquote class="citation">
<p>
Could you let us know what remains to be done to have this ticket completed ?
</p>
</blockquote>
<p>
Nothing special, just do the upgrade.
</p>
TicketjdemeyerWed, 04 Jan 2017 11:03:39 GMTdescription changed; author set; dependencies deleted
https://trac.sagemath.org/ticket/22037#comment:42
https://trac.sagemath.org/ticket/22037#comment:42
<ul>
<li><strong>dependencies</strong>
<em>#19735</em> deleted
</li>
<li><strong>description</strong>
modified (<a href="/ticket/22037?action=diff&version=42">diff</a>)
</li>
<li><strong>author</strong>
set to <em>Jeroen Demeyer</em>
</li>
</ul>
<p>
OK, I'll work on this.
</p>
TicketgitWed, 04 Jan 2017 11:07:33 GMTcommit changed
https://trac.sagemath.org/ticket/22037#comment:43
https://trac.sagemath.org/ticket/22037#comment:43
<ul>
<li><strong>commit</strong>
changed from <em>ba486e12515ad417e2aa747dc5e525e0bedd3d4d</em> to <em>5aeaf5c9f396c297dbd1c1c9b60d0bf5a972e0c9</em>
</li>
</ul>
<p>
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
</p>
<table class="wiki">
<tr><td><a class="extlink" href="https://git.sagemath.org/sage.git/commit/?id=5aeaf5c9f396c297dbd1c1c9b60d0bf5a972e0c9"><span class="icon"></span>5aeaf5c</a></td><td><code>Upgrade to Python 2.7.13</code>
</td></tr></table>
TicketembrayWed, 04 Jan 2017 13:28:32 GMT
https://trac.sagemath.org/ticket/22037#comment:44
https://trac.sagemath.org/ticket/22037#comment:44
<p>
Ah, I think Florent was telling me about this a couple weeks ago. Wish I had been CC'd on thisI've spent a lot of time in Python's class initialization code and probably could have helped. Let me know if I can still help on this.
</p>
<p>
(Ah, I see now the issue was fixed upstream toofunny that Armin raised this issue back in 2009 and *now* it gets fixed :)
</p>
TicketjdemeyerWed, 04 Jan 2017 13:48:54 GMT
https://trac.sagemath.org/ticket/22037#comment:45
https://trac.sagemath.org/ticket/22037#comment:45
<p>
My first impression is that this new version of Python is significantly slower than 2.7.12: I'm seeing several doctest timeouts on a system where I usually don't get any. Granted, my system is loaded from doing things so it might be a false positive. Need to check...
</p>
TicketjdemeyerWed, 04 Jan 2017 13:58:52 GMT
https://trac.sagemath.org/ticket/22037#comment:46
https://trac.sagemath.org/ticket/22037#comment:46
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:44" title="Comment 44">embray</a>:
</p>
<blockquote class="citation">
<p>
Ah, I think Florent was telling me about this a couple weeks ago. Wish I had been CC'd on thisI've spent a lot of time in Python's class initialization code and probably could have helped. Let me know if I can still help on this.
</p>
</blockquote>
<p>
The main question remains: did they break <code>tp_new</code> on other Python versions? The patch got reverted for 2.7.13 but it's not clear whether or not it was applied to some 3.y.x version.
</p>
TicketjdemeyerWed, 04 Jan 2017 14:01:46 GMT
https://trac.sagemath.org/ticket/22037#comment:47
https://trac.sagemath.org/ticket/22037#comment:47
<p>
It worries me that nothing in the Python docs nor in any PEP describes how <code>tp_new</code> is inherited. In my opinion, <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a> makes a significant change which should be subject to a PEP. However, neither the old nor new behaviour is described anywhere. This makes it harder to argue which behaviour is correct.
</p>
TicketembrayWed, 04 Jan 2017 14:13:21 GMT
https://trac.sagemath.org/ticket/22037#comment:48
https://trac.sagemath.org/ticket/22037#comment:48
<p>
Part of the problem is that <code>tp_new</code> is a detail of the CPython API, and while there are PEPs that deal with CPython there aren't many (not enough IMO given that it <em>is</em> the reference implementation). Point being though, this code is very particular to the C API and doesn't particularly impact how Python should work from a pure language design perspective. But I agree, this should be better documented.
</p>
<p>
My guess is just that in the wild there aren't that many handwritten C types outside of Numpy and ones generated by Cython. And it's even less common for them to be used in a multiple inheritance scenario. But just because it isn't common doesn't mean it should be broken or undocumented, so I completely agree. After all this problem was pointed out in 2009 so it's not exactly unheard of.
</p>
TicketembrayWed, 04 Jan 2017 14:26:11 GMT
https://trac.sagemath.org/ticket/22037#comment:49
https://trac.sagemath.org/ticket/22037#comment:49
<p>
FWIW the most relevant PEP to this <a class="extlink" href="https://www.python.org/dev/peps/pep0253/"><span class="icon"></span>https://www.python.org/dev/peps/pep0253/</a>, but I haven't read it in a long time so I forget whether or not it addresses this specific issue, especially given:
</p>
<blockquote class="citation">
<p>
The PEP no longer accurately describes the implementation.
</p>
</blockquote>
TicketjdemeyerWed, 04 Jan 2017 14:31:14 GMT
https://trac.sagemath.org/ticket/22037#comment:50
https://trac.sagemath.org/ticket/22037#comment:50
<p>
Yes, I found that PEP when the discussion about <a class="extlink" href="https://bugs.python.org/issue5322"><span class="icon"></span>https://bugs.python.org/issue5322</a> started but it doesn't answer that specific question.
</p>
TicketembrayWed, 04 Jan 2017 14:35:52 GMT
https://trac.sagemath.org/ticket/22037#comment:51
https://trac.sagemath.org/ticket/22037#comment:51
<p>
Ah, okay. Not surprisingit also says in the PEP:
</p>
<blockquote class="citation">
<p>
While the details of initializing a <a class="missing wiki">PyTypeObject?</a> structure haven't
been documented as such, they are easily gleaned from the examples
in the source code, and I am assuming that the reader is
sufficiently familiar with the traditional way of creating new
Python types in C.
</p>
</blockquote>
<p>
In fairness, those details predate the PEP process in the first place so it's not surprising that there's no PEP for it. But it would be nice to have something better documented in the CAPI docs. It <em>should</em> be part of the API.
</p>
TicketjdemeyerWed, 04 Jan 2017 15:49:27 GMTstatus changed
https://trac.sagemath.org/ticket/22037#comment:52
https://trac.sagemath.org/ticket/22037#comment:52
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_review</em>
</li>
</ul>
TicketcharpentWed, 04 Jan 2017 18:44:58 GMTattachment set
https://trac.sagemath.org/ticket/22037
https://trac.sagemath.org/ticket/22037
<ul>
<li><strong>attachment</strong>
set to <em>python22.7.13.p0.log.gz</em>
</li>
</ul>
<p>
First attempt at installing Python 2.7.13
</p>
TicketcharpentWed, 04 Jan 2017 18:57:19 GMT
https://trac.sagemath.org/ticket/22037#comment:53
https://trac.sagemath.org/ticket/22037#comment:53
<p>
No good, alas... After merging the relevant branch in my current branch (containing R 3.3.2 and the necessary helper packages for xz and pcre), I tried <code>time ./sage i c python2</code>. This took a long time, mostly spent on waiting on failing tests :
</p>
<pre class="wiki">real 48m11,852s
user 6m46,312s
sys 1m12,012s
</pre><p>
At the end of the log file (see attached file), I get :
</p>
<pre class="wiki">353 tests OK.
6 tests failed:
test_gdb test_site test_ssl test_sysconfig test_urllib2
test_urllib2_localnet
1 test altered the execution environment:
test_import
41 tests skipped:
test_aepack test_al test_applesingle test_bsddb test_bsddb185
test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
test_kqueue test_linuxaudiodev test_macos test_macostools
test_msilib test_ossaudiodev test_pep277 test_scriptpackages
test_smtpnet test_socketserver test_startfile test_sunaudiodev
test_timeout test_tk test_ttk_guionly test_unicode_file
test_urllib2net test_urllibnet test_winreg test_winsound
test_zipfile64
4 skips unexpected on linux2:
test_bsddb test_bsddb3 test_dbm test_gdbm
Makefile:877: recipe for target 'test' failed
</pre><p>
Curiously, notwhithstanding the reported problems on the ssl module, pip seems to be able to function somehow :
</p>
<pre class="wiki">charpent@asus16ec:/usr/local/sage7$ sage pip search tralala
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install upgrade pip' command.
charpent@asus16ec:/usr/local/sage7$ sage pip search terminado
terminado (0.6)  Terminals served to term.js using Tornado websockets
INSTALLED: 0.6 (latest)
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install upgrade pip' command.
</pre><p>
(and I get pip advertisements again, source of pseudofailures in tests...)
</p>
<p>
I doubt that it is worth <code>make ptestlong</code> (which monopolizes my machine for a long nice hour, nowadays...).
</p>
<p>
I do not know enough about the specifics of Python in Sage to fel justified in positioning ths ticket as <code>needs_wokj</code>, but I'm sorely tempted.
</p>
<p>
I'll keep my branch in this tate for a while to see what may crop up.
</p>
<p>
HTH,
</p>
TicketjdemeyerWed, 04 Jan 2017 19:19:06 GMT
https://trac.sagemath.org/ticket/22037#comment:54
https://trac.sagemath.org/ticket/22037#comment:54
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:53" title="Comment 53">charpent</a>:
</p>
<blockquote class="citation">
<p>
I do not know enough about the specifics of Python in Sage to fel justified in positioning ths ticket as <code>needs_wokj</code>, but I'm sorely tempted.
</p>
</blockquote>
<p>
Please elaborate. Does this ticket break anything?
</p>
TicketcharpentWed, 04 Jan 2017 19:27:32 GMT
https://trac.sagemath.org/ticket/22037#comment:55
https://trac.sagemath.org/ticket/22037#comment:55
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:54" title="Comment 54">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:53" title="Comment 53">charpent</a>:
</p>
<blockquote class="citation">
<p>
I do not know enough about the specifics of Python in Sage to fel justified in positioning ths ticket as <code>needs_wokj</code>, but I'm sorely tempted.
</p>
</blockquote>
<p>
Please elaborate. Does this ticket break anything?
</p>
</blockquote>
<p>
A cursory, extremely superficial, test(a couple lines in Sage, a bit more in IPython + sympy) didn't show major immediate problems shuch as a crash. But the six bailed test units make me wary.
</p>
<p>
I just launched <code>make ptestlong</code> and I see sage recompiling python2 and brial. Answer in a couple of hours...
</p>
TicketjhpalmieriWed, 04 Jan 2017 20:08:50 GMT
https://trac.sagemath.org/ticket/22037#comment:56
https://trac.sagemath.org/ticket/22037#comment:56
<p>
At some times in the past, the <code>python2</code> package has not passed its test suite. Does the current version pass? That is, are you seeing a regression or the status quo?
</p>
TicketcharpentWed, 04 Jan 2017 21:19:36 GMT
https://trac.sagemath.org/ticket/22037#comment:57
https://trac.sagemath.org/ticket/22037#comment:57
<p>
It might be a regression, but I'm not sure :
</p>
<p>
<code>make ptestlong</code> is currently stalled, with a suspicious number of defunct (zombie) subprocesses. From <code>ps axf</code>, I get (extract) :
</p>
<pre class="wiki">26497 ? Ssl 1:03 \_ /usr/lib/gnometerminal/gnometerminalserver
14353 pts/3 Ss 0:00  \_ bash
20219 pts/3 S+ 0:00   \_ make ptestlong
3447 pts/3 S+ 0:34   \_ python /usr/local/sage7/src/bin/sageruntests p all long logfile=logs/ptestlong.log
3486 pts/3 S+ 0:01   \_ python /usr/local/sage7/src/bin/sagecleaner
15843 pts/3 S 0:01   \_ python /usr/local/sage7/src/bin/sageruntests p all long logfile=logs/ptestlong.log
15844 pts/0 Ssl+ 0:06   \_ /usr/local/sage7/local/bin/giac sage
15845 pts/3 Z 0:00   \_ [python] <defunct>
16021 ? Zs 0:00   \_ [giac] <defunct>
16022 pts/3 Z 0:00   \_ [python] <defunct>
16032 pts/3 Z 0:00   \_ [python] <defunct>
16033 ? Zs 0:00   \_ [giac] <defunct>
16034 pts/3 Z 0:00   \_ [python] <defunct>
16043 pts/3 Z 0:00   \_ [python] <defunct>
17896 pts/4 Ss 0:00  \_ bash
25263 pts/4 R+ 0:00  \_ ps axf
</pre><p>
I'll let it go for a while, and kill it, say, tomorrow morning (it's about 22:05 locally).
</p>
<p>
But this smells. A look at the log (from another terminal) gives me some known bugs, and some new :
</p>
<pre class="wiki">File "src/sage/rings/polynomial/multi_polynomial_ideal.py", line 3532, in sage.r
ings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_basis
Failed example:
ideal(J.transformed_basis()).change_ring(P).interreduced_basis() # optional  giacpy_sage
Expected:
[a  60*c^3 + 158/7*c^2 + 8/7*c  1, b + 30*c^3  79/7*c^2 + 3/7*c, c^4  10/21*c^3 + 1/84*c^2 + 1/84*c]
Got:
[7*a  420*c^3 + 158*c^2 + 8*c  7, 7*b + 210*c^3  79*c^2 + 3*c, 84*c^4  40*c^3 + c^2 + c]
</pre><p>
This one is already known.
</p>
<pre class="wiki">File "src/sage/homology/simplicial_complex.py", line 2812, in sage.homology.simp
licial_complex.SimplicialComplex.is_cohen_macaulay
Failed example:
X.is_cohen_macaulay(ZZ)
Expected:
False
Got:
[Errno 2] No such file or directory: '/home/charpent/.sage/temp/asus16ec/17311/dir_ZhULbV/17421.out'
False
</pre><p>
Might be a race condition : from yet another terminal :
</p>
<pre class="wiki">charpent@asus16ec:/usr/local/sage7$ sage t long warnlong 109.9 src/sage/homology/simplicial_complex.py
Running doctests with ID 201701042208343e41af20.
Git branch: testR
Using optional=database_gap,giacpy_sage,mpir,python2,sage,xz
Doctesting 1 file.
sage t long warnlong 109.9 src/sage/homology/simplicial_complex.py
[588 tests, 3.25 s]

All tests passed!

Total time for all tests: 3.4 seconds
cpu time: 3.0 seconds
cumulative wall time: 3.3 seconds
</pre><pre class="wiki">sage t long warnlong 109.9 src/sage/combinat/root_system/cartan_type.py
Bad exit: 2
**********************************************************************
Tests run before process (pid=17761) failed:
sage: T = CartanType(['A', 4]) ## line 18 ##
sage: T ## line 19 ##
['A', 4]
sage: DynkinDiagram(T) ## line 24 ##
OOOO
1 2 3 4
A4
sage: DynkinDiagram(['A',4]) ## line 33 ##
OOOO
1 2 3 4
A4
sage: DynkinDiagram('A4') ## line 37 ##
OOOO
1 2 3 4
A4
sage: T.dynkin_diagram() ## line 41 ##
OOOO
1 2 3 4
A4
sage: RootSystem(T) ## line 52 ##
Root system of type ['A', 4]
sage: W = WeylGroup(T) ## line 57 ##
</pre><p>
This one is new to me. But again, it might be a race condition :
</p>
<pre class="wiki">charpent@asus16ec:/usr/local/sage7$ sage t long warnlong 109.9 src/sage/combinat/root_system/cartan_type.py
Running doctests with ID 201701042210598ee52f1c.
Git branch: testR
Using optional=database_gap,giacpy_sage,mpir,python2,sage,xz
Doctesting 1 file.
sage t long warnlong 109.9 src/sage/combinat/root_system/cartan_type.py
[446 tests, 1.36 s]

All tests passed!

Total time for all tests: 1.5 seconds
cpu time: 1.1 seconds
cumulative wall time: 1.4 seconds
</pre><p>
And that's all for now. But <code>make ptestlong</code> is still stalled at :
</p>
<pre class="wiki">sage t long warnlong 109.9 local/lib/python2.7/sitepackages/sagenb/testing/selenium/__init__.py
[0 tests, 0.00 s]
sage t long warnlong 109.9 src/sage/interfaces/giac.py
</pre><p>
If you feel it useful, I'll attach the resulting ptestlong.log after tomorrow's <code>kill</code>.
</p>
TicketcharpentWed, 04 Jan 2017 21:23:50 GMT
https://trac.sagemath.org/ticket/22037#comment:58
https://trac.sagemath.org/ticket/22037#comment:58
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:56" title="Comment 56">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
At some times in the past, the <code>python2</code> package has not passed its test suite. Does the current version pass?
</p>
</blockquote>
<p>
No. Six test units failed (see above) .
</p>
<blockquote class="citation">
<p>
That is, are you seeing a regression or the status quo?
</p>
</blockquote>
<p>
I don't know if these test units used to pass before...
</p>
<p>
HTH,
</p>
TicketcharpentWed, 04 Jan 2017 21:49:13 GMTattachment set
https://trac.sagemath.org/ticket/22037
https://trac.sagemath.org/ticket/22037
<ul>
<li><strong>attachment</strong>
set to <em>ptestlong.log.gz</em>
</li>
</ul>
<p>
result of make ptestlong, first attempt on 2.7.13
</p>
TicketcharpentWed, 04 Jan 2017 21:57:58 GMT
https://trac.sagemath.org/ticket/22037#comment:59
https://trac.sagemath.org/ticket/22037#comment:59
<p>
New data point : after about 1 hour, the test of <code>giac</code> inteface timed out, liberating the whole thing (the zombie processes disappeared).
</p>
<p>
The resulting <code>ptestlong.log</code> has been attached, just in case...
</p>
<p>
Interestingly, this may be a race condition : the very same test passes standalone in about 1.33 seconds :
</p>
<pre class="wiki">charpent@asus16ec:/usr/local/sage7$ sage t long warnlong 109.9 src/sage/interfaces/giac.py
Running doctests with ID 20170104225102b275d916.
Git branch: testR
Using optional=database_gap,giacpy_sage,mpir,python2,sage,xz
Doctesting 1 file.
sage t long warnlong 109.9 src/sage/interfaces/giac.py
[157 tests, 1.33 s]

All tests passed!

Total time for all tests: 6.4 seconds
cpu time: 0.6 seconds
cumulative wall time: 1.3 seconds
</pre><p>
So we have two, possibly three problems :
</p>
<ul><li>failed test units in Python's own test suite ;
</li><li>transient errors (race conditions) in <code>ptestlong</code> ;
</li><li>an unfixed known "error" (not mathematically an error, bit, asufar as I understand, a different expression of the same result).
</li></ul><p>
Advice ?
</p>
TicketjhpalmieriWed, 04 Jan 2017 23:27:21 GMT
https://trac.sagemath.org/ticket/22037#comment:60
https://trac.sagemath.org/ticket/22037#comment:60
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:58" title="Comment 58">charpent</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:56" title="Comment 56">jhpalmieri</a>:
</p>
<blockquote class="citation">
<p>
At some times in the past, the <code>python2</code> package has not passed its test suite. Does the current version pass?
</p>
</blockquote>
<p>
No. Six test units failed (see above) .
</p>
</blockquote>
<p>
I meant, do the unit tests pass for 2.7.10 (= the current version)?
</p>
TicketcharpentThu, 05 Jan 2017 01:39:42 GMT
https://trac.sagemath.org/ticket/22037#comment:61
https://trac.sagemath.org/ticket/22037#comment:61
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:60" title="Comment 60">jhpalmieri</a>:
</p>
<p>
[ Snip...]
</p>
<blockquote class="citation">
<p>
I meant, do the unit tests pass for 2.7.10 (= the current version)?
</p>
</blockquote>
<p>
Current is 2.7.12 (since 7.5rc0).
</p>
<pre class="wiki">354 tests OK.
5 tests failed:
test_gdb test_site test_sysconfig test_urllib2
test_urllib2_localnet
42 tests skipped:
test_aepack test_al test_applesingle test_bsddb test_bsddb185
test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
test_kqueue test_linuxaudiodev test_macos test_macostools
test_msilib test_ossaudiodev test_pep277 test_scriptpackages
test_smtpnet test_socketserver test_ssl test_startfile
test_sunaudiodev test_timeout test_tk test_ttk_guionly
test_unicode_file test_urllib2net test_urllibnet test_winreg
test_winsound test_zipfile64
5 skips unexpected on linux2:
test_bsddb test_bsddb3 test_dbm test_gdbm test_ssl
</pre><p>
So there is more lossage (<code>test_ssl</code> passed in 2.7.12).
</p>
TicketcharpentThu, 05 Jan 2017 02:19:37 GMT
https://trac.sagemath.org/ticket/22037#comment:62
https://trac.sagemath.org/ticket/22037#comment:62
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/22037#comment:61" title="Comment 61">charpent</a>:
</p>
<p>
Going back to 7.5beta6 (which had 2.7.10, I get :
</p>
<pre class="wiki">4 tests failed:
test_gdb test_site test_sysconfig test_urllib2_localnet
42 tests skipped:
test_aepack test_al test_applesingle test_bsddb test_bsddb185
test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
test_dbm test_dl test_gdbm test_gl test_imageop test_imgfile
test_kqueue test_linuxaudiodev test_macos test_macostools
test_msilib test_ossaudiodev test_pep277 test_scriptpackages
test_smtpnet test_socketserver test_ssl test_startfile
test_sunaudiodev test_timeout test_tk test_ttk_guionly
test_unicode_file test_urllib2net test_urllibnet test_winreg
test_winsound test_zipfile64
5 skips unexpected on linux2:
test_bsddb test_bsddb3 test_dbm test_gdbm test_ssl
</pre><p>
So test_urllib2 was lost between 2.7.10 and 2.7.12. And, BTW, test_ssl didn't pass in 2.7.12 : it was marked as unexpected.
</p>
<p>
HTH,
</p>
TicketjdemeyerThu, 05 Jan 2017 07:05:29 GMT
https://trac.sagemath.org/ticket/22037#comment:63
https://trac.sagemath.org/ticket/22037#comment:63
<p>
There is some significant speed regression going from 2.7.12 to 2.7.13.
</p>
<p>
I doctested the directory <code>src/sage/manifolds</code> 3 times with each Python version on an otherwise idle system.
</p>
<p>
With 2.7.12, I got
</p>
<pre class="wiki">manifolds2.7.12.log:Total time for all tests: 619.3 seconds
manifolds2.7.12.log:Total time for all tests: 625.6 seconds
manifolds2.7.12.log:Total time for all tests: 616.8 seconds
</pre><p>
With 2.7.13:
</p>
<pre class="wiki">manifolds2.7.13.log:Total time for all tests: 645.1 seconds
manifolds2.7.13.log:Total time for all tests: 640.7 seconds
manifolds2.7.13.log:Total time for all tests: 646.0 seconds
</pre><p>
I don't know what to think of this...
</p>
TicketjdemeyerSat, 07 Jan 2017 11:46:38 GMTpriority changed
https://trac.sagemath.org/ticket/22037#comment:64
https://trac.sagemath.org/ticket/22037#comment:64
<ul>
<li><strong>priority</strong>
changed from <em>critical</em> to <em>blocker</em>
</li>
</ul>
<p>
Blocker because of <a class="closed ticket" href="https://trac.sagemath.org/ticket/22147" title="defect: pyport.h from python2.7 sometimes clashes with c++ functions on OS X (closed: fixed)">#22147</a>.
</p>
TicketvbraunSat, 07 Jan 2017 14:30:49 GMTstatus changed; reviewer set
https://trac.sagemath.org/ticket/22037#comment:65
https://trac.sagemath.org/ticket/22037#comment:65
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
set to <em>Volker Braun</em>
</li>
</ul>
<p>
Since we need this to compile on OSX...
</p>
<p>
remaining issues should be moved to separate tickets.
</p>
TicketvbraunSun, 08 Jan 2017 00:54:39 GMTstatus, branch changed; resolution set
https://trac.sagemath.org/ticket/22037#comment:66
https://trac.sagemath.org/ticket/22037#comment:66
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
<li><strong>branch</strong>
changed from <em>u/jdemeyer/upgrade_to_python_2_7_13</em> to <em>5aeaf5c9f396c297dbd1c1c9b60d0bf5a972e0c9</em>
</li>
</ul>
Ticket