py3: make all tests pass

[chapoton]

This ticket is used to keep track of the progress towards python3.

release	files having failing doctests	more than 100	from 10 to 100	from 1 to 10	failing doctests
8.4.b3	915	42	251	622	17972
8.4.b4	803	23	175	605	10921
8.4.b5	657	17	145	495	8509
8.4.b6	594	17	127	450	7706
8.4.b7	587	17	128	442	7678
8.4.rc0	524	16	119	389	6976
8.5.b0	461	6	94	361	4206
8.5.b1	414	3	76	335	3206
8.5.b2	370	1	66	303	2465
8.5.b3	339	1	62	276	2245
8.5.b4	312	0	58	254	1947
8.5.b5	295	0	51	244	1741
8.5.b6	264	0	43	221	1514
8.5.rc0	252	0	39	213	1393
8.6.b0	234	0	30	204	1163
8.6.b1	231	0	30	201	1146
8.6.rc0	232	0	28	204	1124
8.6.rc1	236	0	30	206	1136

Folders where all tests pass (8.6.rc0):

• algebras/steenrod ✔ #26219 and #26220
• arith #26208 ✔
• coding #26026 #26765 (final)
• combinat/rigged_configurations #26805
• combinat/sf (#26220 +) #26215
• combinat/words #26362, #26458, #26473, #26590
• databases #26702 #26704
• docs ✔ #26310
• features ✔
• finance #26213 ✔
• games ✔
• game_theory #26429 #26738 (final)
• groups/lie_gps #26518
• knots ✔
• libs/gsl
• lfunctions ✔
• logic #26214 ✔
• manifolds #26475
• media ✔
• monoids ✔ #26294
• parallel ✔
• probability #26216 ✔
• quadratic_forms/ #26224 ✔
• quivers #26211 ✔
• rings/asymptotic #26146
• rings/number_field #26263
• rings/function_field #26721 #26764 (final)
• sat #26543 ✔
• schemes/toric #26664
• server ✔
• stats #26401 ✔
• structure #25694 #26768
• tensor #26295, #26601
• typeset ✔

Folders with work in progress:

• algebras #26316 #26399 #26279 #26646 #26688 #26766
• calculus #26751 #26988 (final)
• categories #26545 #26780
• combinat/designs #26430, #26837 (regression)
• combinat/finite_state_machine #26147, #26468, #26737
• combinat/posets #26998 (final)
• combinat/root_system #26649
• combinat/species #26328, #26610
• combinat/subset #26510
• combinat #26535 #26671 #26719 #26778 #26781 #26917 #26918
• cpython #26855
• crypto #26398
• data_structures #24852
• doc/en #26346
• functions #24758, #26591 #26865 #26770 #25979
• geometry #26726
• graphs #26431 #26447 #26527 #26640 #26645 #26846
• groups #26799
• groups/matrix_gps #26400 (regression)
• interfaces #26314
• libs #24804 #26665
• matrix/ #26236 #26716
• matroids #26767
• misc #26312, #26390 #26674 #26720 #27001
• modular #26382, #26611, #26967 #25301
• modular/modsym #26673
• modules #26276
• numerical/backends #24740 (for char * arguments management) and #24741, #26631
• plot #26210 (partial fix only) #26299 #26301 #26321
• repl #26881
• rings #26697
• rings (power series) #26539
• rings/polynomial #26670 #26760 #26909
• rings/number_field #26619
• sandpiles #26016 #26628 #26722
• sets #26659
• schemes/elliptic_curves #26536, #26529, #26570, #26594 #26913 #26914
• tests/french_book #26495 #23572

[embray]

Hadn't even seen this ticket; very nice.

[chapoton]

good progress on 8.5.b0

[chapoton]

The six remaining files with many failures are

['sage -t --long src/sage/combinat/finite_state_machine.py  # 189 doctests failed',
 'sage -t --long src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # 105 doctests failed',
 'sage -t --long src/sage/graphs/generic_graph.py  # 110 doctests failed',
 'sage -t --long src/sage/groups/lie_gps/nilpotent_lie_group.py  # 143 doctests failed',
 'sage -t --long src/sage/rings/padics/padic_lattice_element.py  # 167 doctests failed',
 'sage -t --long src/sage/rings/valuation/limit_valuation.py  # 106 doctests failed']

[jhpalmieri]

For whoever wants to work on it, regarding

sage -t --long src/sage/combinat/finite_state_machine.py  # 189 doctests failed

about 150 of the failures can be cured by replacing 'iteritems' by 'items' in lines like

        elif hasattr(data, 'iteritems'):
            # data is a dict (or something similar),

I don't know if 'items' is the right attribute to check, or if it would actually be better to change to isinstance(data, dict). What other data structures are mean to be acceptable? What is "something similar" to a dict?

[vklein]

Replying to jhpalmieri:

For whoever wants to work on it, regarding ... I don't know if 'items' is the right attribute to check, or if it would actually be better to change to isinstance(data, dict). What other data structures are mean to be acceptable? What is "something similar" to a dict?

There is some custom "Mapping types" in sage like FreeModuleElement_generic_sparse, which implements items and iteritems but doesn't inherit from dict.

[chapoton]

tool to extract the data from patchbot log

[chapoton]

There was a regression in the manifolds folder (sigh)..

The 3 remaining files with more than 100 failing doctests are

['sage -t --long src/sage/combinat/finite_state_machine.py  # 182 doctests failed',
 'sage -t --long src/sage/graphs/generic_graph.py  # 109 doctests failed',
 'sage -t --long src/sage/rings/padics/padic_lattice_element.py  # 167 doctests failed']

The first one is already tackled in #26147 at least partially.

[jhpalmieri]

Regarding

'sage -t --long src/sage/rings/padics/padic_lattice_element.py  # 167 doctests failed'

I can reproduce those failures when I run ./sage -t ..., but (at least for the first few) not when I run Sage interactively. Odd.

[chapoton]

Top ten bad files in 8.5.b2 are

['sage -t --long src/sage/numerical/backends/cvxopt_sdp_backend.pyx  # 39 doctests failed',
 'sage -t --long src/sage/algebras/iwahori_hecke_algebra.py  # 40 doctests failed',
 'sage -t --long src/sage/numerical/mip.pyx  # 47 doctests failed',
 'sage -t --long src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 'sage -t --long src/sage/rings/function_field/ideal.py  # 56 doctests failed',
 'sage -t --long src/sage/graphs/graph.py  # 59 doctests failed',
 'sage -t --long src/sage/graphs/connectivity.pyx  # 60 doctests failed',
 'sage -t --long src/sage/misc/explain_pickle.py  # 70 doctests failed',
 'sage -t --long src/sage/numerical/sdp.pyx  # 75 doctests failed',
 'sage -t --long src/sage/graphs/generic_graph.py  # 109 doctests failed']

[chapoton]

The top ten worse files in 8.5.b3 are

['sage -t --long src/sage/combinat/finite_state_machine.py  # 31 doctests failed',
 'sage -t --long src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed',
 'sage -t --long src/sage/numerical/mip.pyx  # 47 doctests failed',
 'sage -t --long src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 'sage -t --long src/sage/rings/function_field/ideal.py  # 56 doctests failed',
 'sage -t --long src/sage/graphs/graph.py  # 58 doctests failed',
 'sage -t --long src/sage/graphs/connectivity.pyx  # 59 doctests failed',
 'sage -t --long src/sage/rings/tate_algebra_element.pyx  # 63 doctests failed',
 'sage -t --long src/sage/misc/explain_pickle.py  # 70 doctests failed',
 'sage -t --long src/sage/graphs/generic_graph.py  # 115 doctests failed']

[chapoton]

Top ten worst files in 8.5.b4 are:

src/sage/graphs/graph.py  # 30 doctests failed
src/sage/numerical/linear_functions.pyx  # 30 doctests failed
src/sage/combinat/finite_state_machine.py  # 31 doctests failed
src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed
src/sage/numerical/mip.pyx  # 47 doctests failed
src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed
src/sage/graphs/connectivity.pyx  # 50 doctests failed
src/sage/rings/function_field/ideal.py  # 56 doctests failed
src/sage/misc/explain_pickle.py  # 70 doctests failed
src/sage/graphs/generic_graph.py  # 77 doctests failed

[chapoton]

Number of failing doctests in 8.5.beta4 by components

'en': 56, 'coding': 28, 'data_structures': 5, 'symbolic': 24, 'libs': 34,
 'schemes': 102, 'combinat': 292, 'dynamics': 39, 'interacts': 6, 'homology': 29,
 'plot': 5, 'functions': 5, 'matrix': 16, 'rings': 209, 'categories': 34,
 'docbuild': 2, 'algebras': 21, 'common': 1, 'calculus': 5, 'tests': 40,
 'matroids': 75, 'misc': 172, 'interfaces': 1, 'numerical': 206, 'modular': 82,
 'repl': 13, 'groups': 14, 'sandpiles': 22, 'game_theory': 11, 'structure': 6,
 'cpython': 7, 'sets': 6, 'geometry': 32, 'modules': 3, 'crypto': 13,
 'graphs': 292, 'ext': 8, 'doctest': 8, 'databases': 16

[chapoton]

Worst 15 files in 8.5.b5:

[' src/sage/homology/simplicial_complex.py  # 23 doctests failed',
 ' src/sage/numerical/linear_tensor_element.pyx  # 23 doctests failed',
 ' src/sage/matroids/matroid.pyx  # 24 doctests failed',
 ' src/sage/rings/integer.pyx  # 24 doctests failed',
 ' src/sage/misc/decorators.py  # 28 doctests failed',
 ' src/sage/numerical/linear_tensor_constraints.py  # 28 doctests failed',
 ' src/sage/combinat/rigged_configurations/kr_tableaux.py  # 30 doctests failed',
 ' src/sage/graphs/graph.py  # 30 doctests failed',
 ' src/sage/numerical/linear_functions.pyx  # 30 doctests failed',
 ' src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed',
 ' src/sage/numerical/mip.pyx  # 47 doctests failed',
 ' src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 ' src/sage/graphs/connectivity.pyx  # 49 doctests failed',
 ' src/sage/misc/explain_pickle.py  # 70 doctests failed',
 ' src/sage/graphs/generic_graph.py  # 79 doctests failed']

[chapoton]

and by components:

[('common', 1), ('interfaces', 1), ('docbuild', 2), ('coding', 2), ('modules', 4), 
('functions', 5), ('calculus', 5), ('plot', 5), ('data_structures', 5), ('structure',
 6), ('interacts', 6), ('sets', 6), ('cpython', 7), ('doctest', 8), ('ext', 8), 
('crypto', 13), ('geometry', 13), ('repl', 13), ('groups', 15), ('algebras', 16),
 ('matrix', 17), ('sandpiles', 22), ('symbolic', 26), ('homology', 30), ('libs', 34),
 ('categories', 35), ('tests', 40), ('dynamics', 40), ('en', 56), ('matroids', 73),
 ('schemes', 80), ('modular', 82), ('rings', 141), ('misc', 171), ('numerical', 206), 
('combinat', 240), ('graphs', 300)]

[chapoton]

worst 20 files in 8.5.b6:

['/sage/matroids/matroid.pyx  # 18 doctests failed',
 '/sage/combinat/finite_state_machine.py  # 20 doctests failed',
 '/sage/misc/classcall_metaclass.pyx  # 20 doctests failed',
 '/sage/numerical/linear_tensor.py  # 20 doctests failed',
 '/sage/dynamics/arithmetic_dynamics/wehlerK3.py  # 21 doctests failed',
 '/sage/rings/polynomial/multi_polynomial_ideal.py  # 21 doctests failed',
 '/sage/homology/simplicial_complex.py  # 22 doctests failed',
 '/sage/sandpiles/sandpile.py  # 22 doctests failed',
 '/sage/numerical/linear_tensor_element.pyx  # 23 doctests failed',
 '/sage/rings/integer.pyx  # 24 doctests failed',
 '/sage/misc/decorators.py  # 28 doctests failed',
 '/sage/numerical/linear_tensor_constraints.py  # 28 doctests failed',
 '/sage/graphs/graph.py  # 30 doctests failed',
 '/sage/numerical/linear_functions.pyx  # 30 doctests failed',
 '/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed',
 '/sage/numerical/mip.pyx  # 47 doctests failed',
 '/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 '/sage/graphs/connectivity.pyx  # 49 doctests failed',
 '/sage/misc/explain_pickle.py  # 70 doctests failed',
 '/sage/graphs/generic_graph.py  # 78 doctests failed']

[chapoton]

15 worst offenders in 8.5.rc0:

['src/sage/dynamics/arithmetic_dynamics/wehlerK3.py  # 21 doctests failed',
 'src/sage/matroids/matroid.pyx  # 21 doctests failed',
 'src/sage/rings/polynomial/multi_polynomial_ideal.py  # 21 doctests failed',
 'src/sage/homology/simplicial_complex.py  # 22 doctests failed',
 'src/sage/sandpiles/sandpile.py  # 22 doctests failed',
 'src/sage/numerical/linear_tensor_element.pyx  # 23 doctests failed',
 'src/sage/misc/decorators.py  # 28 doctests failed',
 'src/sage/numerical/linear_tensor_constraints.py  # 28 doctests failed',
 'src/sage/graphs/graph.py  # 30 doctests failed',
 'src/sage/numerical/linear_functions.pyx  # 30 doctests failed',
 'src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed',
 'src/sage/numerical/mip.pyx  # 47 doctests failed',
 'src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 'src/sage/misc/explain_pickle.py  # 70 doctests failed',
 'src/sage/graphs/generic_graph.py  # 74 doctests failed']

[chapoton]

15 worst offenders in 8.6.b0:

['src/sage/categories/finite_posets.py  # 15 doctests failed',
 'src/sage/combinat/matrices/hadamard_matrix.py  # 15 doctests failed',
 'src/sage/combinat/finite_state_machine.py  # 19 doctests failed',
 'src/sage/misc/classcall_metaclass.pyx  # 20 doctests failed',
 'src/sage/dynamics/arithmetic_dynamics/wehlerK3.py  # 21 doctests failed',
 'src/sage/rings/polynomial/multi_polynomial_ideal.py  # 21 doctests failed',
 'src/sage/sandpiles/sandpile.py  # 22 doctests failed',
 'src/sage/matroids/matroid.pyx  # 23 doctests failed',
 'src/sage/homology/simplicial_complex.py  # 24 doctests failed',
 'src/sage/graphs/graph.py  # 28 doctests failed',
 'src/sage/misc/decorators.py  # 28 doctests failed',
 'src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed',
 'src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 'src/sage/graphs/generic_graph.py  # 68 doctests failed',
 'src/sage/misc/explain_pickle.py  # 70 doctests failed']

[chapoton]

maybe due to the upgrade of gap, there is a regression:

sage -t --long src/sage/groups/matrix_gps/finitely_generated.py
  # 3 doctests failed

[chapoton]

worst 15 files in 8.6.b1:

['src/sage/categories/finite_posets.py  # 15 doctests failed',
 'src/sage/combinat/matrices/hadamard_matrix.py  # 15 doctests failed',
 'src/sage/matroids/matroid.pyx  # 17 doctests failed',
 'src/sage/combinat/finite_state_machine.py  # 19 doctests failed',
 'src/sage/misc/classcall_metaclass.pyx  # 20 doctests failed',
 'src/sage/dynamics/arithmetic_dynamics/wehlerK3.py  # 21 doctests failed',
 'src/sage/rings/polynomial/multi_polynomial_ideal.py  # 21 doctests failed',
 'src/sage/sandpiles/sandpile.py  # 22 doctests failed',
 'src/sage/homology/simplicial_complex.py  # 23 doctests failed',
 'src/sage/graphs/graph.py  # 28 doctests failed',
 'src/sage/misc/decorators.py  # 28 doctests failed',
 'src/sage/schemes/riemann_surfaces/riemann_surface.py  # 36 doctests failed',
 'src/doc/en/thematic_tutorials/sandpile.rst  # 49 doctests failed',
 'src/sage/graphs/generic_graph.py  # 63 doctests failed',
 'src/sage/misc/explain_pickle.py  # 70 doctests failed']

[chapoton]

files where failing doctests have re-appeared (regressions):

['src/sage/categories/crystals.py',
 'src/sage/matroids/circuit_closures_matroid.pyx',
 'src/sage/matroids/dual_matroid.py']

file where failing doctests have disappeared (progress):

['src/sage/misc/decorators.py',
 'src/sage/rings/polynomial/laurent_polynomial.pyx']

so this is a net loss of 1 file

[chapoton]

Getting worse again! Damn!