Python 3 preparation: Adapt to the changed integer division semantics (/ vs. //)

[wluebbe]

See original ticket below. We can still use this as a meta-ticket for tracking all division-related changes needed for Python 3. Existing tickets include:

Progress Report

Open

No results

In Progress

No results

Done

#15995

Python 3 preparation: Adapt to the changed integer division semantics (/ vs. //)

#20533

Change again integer divisions from / to // in combinat

#21127

py3: turn some divisions in compatible format

#22767

py3: some care for division

#23729

py3: some care for division in graphs + various improvements

#23731

py3: future division in matrix2

#24208

py3: add some future division imports

#24805

py3: miscellaneous division-related fixes, particularly for sage_setup.autogen

#25180

py3: fixes to sage.dynamics

#26135

py3: fix bad divisions in givaro finite fields

#29139

*.pxi files might not be Python3-ready

Modules needing work

Based on some grepping results, the following modules appear to contain a class that implements __div__ but not __truediv__ (the latter should be preferred in all cases):

Ticket	Module
#?????	src/sage/algebras/lie_algebras/lie_algebra_element.pyx
#?????	src/sage/categories/examples/lie_algebras.py
#24805	src/sage/ext/fast_callable.pyx (this is just from a Python 2 only test case from the relevant ticket)
#?????	src/sage/groups/additive_abelian/qmodnz_element.py
#?????	src/sage/matrix/matrix_gfpn_dense.pyx
#?????	src/sage/rings/integer.pyx
#?????	src/sage/rings/rational.pyx
#?????	src/sage/structure/category_object.pyx

---

The new semantics can be enabled in Python 2 with from __future__ import division.

This will require checking where to replace / by //

Changes according to libfuturize/fixes/division.py:

Add
from __future__ import division

This ticket is tracked as a dependency of meta-ticket ticket:15980.

To test what remains to be done, one can use:

export PYTHONIOENCODING="utf-8"
source ./local/bin/sage-env
python -Qnew ./local/bin/sage-runtests --all

[wluebbe]

The first attempt went like this: futurize -w -f division src/.
This added from __future__ import division to 411 py modules.
Running the doctests ./sage -t -p --all gave billions of lines of doctest failures ... - I had to cancel the run.
But this result should not have come as a surprise because I had not fixed anything yet ...

The task is to identify those / and /= that operate solely on ints and longs and then change them to // and //=.

The Python source distribution has a script Tools/scripts/finddiv.py which has identified similar 416 py modules as the above programs (it performs clearly better as a simple grep since it tokenizes the code). It also outputs those 2625 lines that contain the / and /= operators - but for all types of operands (the analysis is purely syntactical).

There is another script Tools/scripts/fixdiv.py that uses the output of
python -Qwarnall yourscripts 2>the-warnings to create a simple (commented) patch file for those division operators that only deal with ints and longs. This is possible since it uses the type information obtained by running the Python programs. This should be much more useful!

But currently I am stuck:

• I don't know how to pass the option -Qwarnall to the Python interpreter when running the doctests
• and the warning output screws up the doctests.

It may be possible to use the warnings module to write the warnings to a file instead if stderr (by replacing warnings.showwarning() with a custom function).

[wluebbe]

I finally manage to run the doctests with the Python command line option -3. This is how I did it:

...:~/sage-6.2.beta8$ . src/bin/sage-env
...:~/sage-6.2.beta8$ python -3 ./local/bin/sage-runtests  src/sage 1>logs/sage-warnings3-stdout.txt 2>logs/sage-warnings3-stderr.txt

A summary of stderr output is

nbr_lines= 133
   83  Overriding __eq__ blocks inheritance of __hash__ in 3.x
   27  classic int division
   18  type inequality comparisons not supported in 3.x
    2  the cmp argument is not supported in 3.x
    1  execfile() not supported in 3.x; use exec()
    1  file.softspace not supported in 3.x
    1  tmpfile has been removed in 3.x; use the tempfile module

and of stdout

nbr_lines= 39630
19386  type inequality comparisons not supported in 3.x
 8779  Overriding __eq__ blocks inheritance of __hash__ in 3.x
 4039  CObject type is not supported in 3.x. Please use capsule objects instead.
 2139  reduce() not supported in 3.x; use functools.reduce()
 2017  classic int division
 2000  the cmp argument is not supported in 3.x
  789  comparing unequal types not supported in 3.x
  262  __getitem__ not supported for exception classes in 3.x; use args attribute
   88  sys.exc_clear() not supported in 3.x; use except clauses
   57  in 3.x, __getslice__ has been removed; use __getitem__
   14  file.softspace not supported in 3.x
   12  the commands module has been removed in Python 3.0; use the subprocess module instead
    7  execfile() not supported in 3.x; use exec()
    7  tmpfile has been removed in 3.x; use the tempfile module
    5  dict inequality comparisons not supported in 3.x
    3  
    2  Use the Permutations object instead.
    2  This class is replaced by Matrix_modn_dense_float/Matrix_modn_dense_double.
    2  to_permutation() is deprecated. Use instead reading_word_permutation()
    2  this class is deprecated. Use RibbonShapedTableau instead
    2  BaseException.message has been deprecated as of Python 2.6
    2  The 'new' module has been removed in Python 3.0; use the 'types' module instead.
    2  this is done automatically by the doctest framework
    2  derangements() is deprecated. Use Derangements instead.
    1  The output of Transducer.cartesian_product will change. Please use Transducer.intersection for the original output.
    1  Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you can use named arguments instead, like EXPR(x=..., y=...)
    1  This method is deprecated. Use triangulate() instead.
    1  In 3.x, reload() is renamed to imp.reload()
    1  dict.has_key() not supported in 3.x; use the in operator
    1  The output of
    1  Substitution using
    1  This method is deprecated. Use triangulate().simplicial_complex() instead.
    1  Variable of integration should be specified explicitly.
    1  OrderedAlphabet is deprecated; use Alphabet instead.

Unfortunately typical stdout printout looks like

**********************************************************************
File "src/sage/modular/modsym/space.py", line 929, in sage.modular.modsym.space.ModularSymbolsSpace._q_expansion_module
Failed example:
    ModularSymbols(11, 2, base_ring=GF(4,'a')).cuspidal_submodule()._q_expansion_module(prec=4, algorithm="hecke")
Expected:
    Vector space of degree 4 and dimension 1 over Finite Field in a of size 2^2
    Basis matrix:
    [0 1 0 1]
Got:
    doctest:127: DeprecationWarning: type inequality comparisons not supported in 3.x
    doctest:511: DeprecationWarning: classic int division
    doctest:250: DeprecationWarning: type inequality comparisons not supported in 3.x
    doctest:585: DeprecationWarning: classic int division
    doctest:286: DeprecationWarning: classic int division
    doctest:396: DeprecationWarning: classic int division
    doctest:6464: DeprecationWarning: classic int division
    Vector space of degree 4 and dimension 1 over Finite Field in a of size 2^2
    Basis matrix:
    [0 1 0 1]
**********************************************************************

and I don't know how to correlate line likes doctest:511 with the original line.

The line in stderr are more useful, but are few and far between

~/sage-6.2.beta8/local/lib/python2.7/site-packages/sage/groups/perm_gps/cubegroup.py:368: DeprecationWarning: classic int division
  2:  [1/2,3/2,5/2], #  ub

[aapitzsch]

Hi Wilfried, in #16371 I changed some / to //, maybe you can review it.

BTW: how you got the summary of stdout and stderr?

[wluebbe]

Small Python script to create the summary

[wluebbe]

For the summary see the small attached Py script ;-)

I will look at your ticket.

[wluebbe]

I created the ticket #18578 to address classes which define the special function __div__() (and __truediv__()). See also the comments 5 and 7 in #16731.

[jdemeyer]

One very small effort in the spirit of this ticket: #19535

[jdemeyer]

Also #19536.

[tmonteil]

Replying to wluebbe:

But currently I am stuck:

• I don't know how to pass the option -Qwarnall to the Python interpreter when running the doctests

Could you try the following:

From a terminal, run sage -sh

From this shell, run:

export PYTHONIOENCODING="utf-8"
python -Qwarnall $(which sage-runtests) <options>

Where <options> is something like --all --long (depending on the options you are using for doctesting) ?

[chapoton]

one step done in #21127

[chapoton]

another step in #22767

[chapoton]

another step in #23731

[chapoton]

another step in #24548

[embray]

Some more status tracking for division-related work.

[chapoton]

updated script

[chapoton]

I think that we are now ready to close this one. Agreed ?