Opened 7 years ago

Closed 16 months ago

#15995 closed enhancement (fixed)

Python 3 preparation: Adapt to the changed integer division semantics (/ vs. //)

Reported by: wluebbe Owned by: embray
Priority: major Milestone: sage-duplicate/invalid/wontfix
Component: python3 Keywords: division
Cc: embray, jdemeyer, fbissey, tscrim, jhpalmieri, vklein Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by embray)

See original ticket below. We can still use this as a meta-ticket for tracking all division-related changes needed for Python 3. Existing tickets include:

Progress Report

Open

No results

In Progress

No results

Done

#15995
Python 3 preparation: Adapt to the changed integer division semantics (/ vs. //)
#20533
Change again integer divisions from / to // in combinat
#21127
py3: turn some divisions in compatible format
#22767
py3: some care for division
#23729
py3: some care for division in graphs + various improvements
#23731
py3: future division in matrix2
#24208
py3: add some future division imports
#24805
py3: miscellaneous division-related fixes, particularly for sage_setup.autogen
#25180
py3: fixes to sage.dynamics
#26135
py3: fix bad divisions in givaro finite fields
#29139
*.pxi files might not be Python3-ready

Modules needing work

Based on some grepping results, the following modules appear to contain a class that implements __div__ but not __truediv__ (the latter should be preferred in all cases):

Ticket Module
#????? src/sage/algebras/lie_algebras/lie_algebra_element.pyx
#????? src/sage/categories/examples/lie_algebras.py
#24805 src/sage/ext/fast_callable.pyx
(this is just from a Python 2 only test case from the relevant ticket)
#????? src/sage/groups/additive_abelian/qmodnz_element.py
#????? src/sage/matrix/matrix_gfpn_dense.pyx
#????? src/sage/rings/integer.pyx
#????? src/sage/rings/rational.pyx
#????? src/sage/structure/category_object.pyx

The new semantics can be enabled in Python 2 with from __future__ import division.

This will require checking where to replace / by //

Changes according to libfuturize/fixes/division.py:

Add
from __future__ import division

This ticket is tracked as a dependency of meta-ticket ticket:15980.

To test what remains to be done, one can use:

export PYTHONIOENCODING="utf-8"
source ./local/bin/sage-env
python -Qnew ./local/bin/sage-runtests --all

Attachments (2)

read_deprecation-warnings-v1b.py (931 bytes) - added by wluebbe 6 years ago.
Small Python script to create the summary
read_deprecation-warnings-v2.py (908 bytes) - added by chapoton 2 years ago.
updated script

Download all attachments as: .zip

Change History (26)

comment:1 follow-up: Changed 6 years ago by wluebbe

The first attempt went like this: futurize -w -f division src/.
This added from __future__ import division to 411 py modules.
Running the doctests ./sage -t -p --all gave billions of lines of doctest failures ... - I had to cancel the run.
But this result should not have come as a surprise because I had not fixed anything yet ...

The task is to identify those / and /= that operate solely on ints and longs and then change them to // and //=.

The Python source distribution has a script Tools/scripts/finddiv.py which has identified similar 416 py modules as the above programs (it performs clearly better as a simple grep since it tokenizes the code). It also outputs those 2625 lines that contain the / and /= operators - but for all types of operands (the analysis is purely syntactical).

There is another script Tools/scripts/fixdiv.py that uses the output of
python -Qwarnall yourscripts 2>the-warnings to create a simple (commented) patch file for those division operators that only deal with ints and longs. This is possible since it uses the type information obtained by running the Python programs. This should be much more useful!

But currently I am stuck:

  • I don't know how to pass the option -Qwarnall to the Python interpreter when running the doctests
  • and the warning output screws up the doctests.

It may be possible to use the warnings module to write the warnings to a file instead if stderr (by replacing warnings.showwarning() with a custom function).

comment:2 Changed 6 years ago by wluebbe

I finally manage to run the doctests with the Python command line option -3. This is how I did it:

...:~/sage-6.2.beta8$ . src/bin/sage-env
...:~/sage-6.2.beta8$ python -3 ./local/bin/sage-runtests  src/sage 1>logs/sage-warnings3-stdout.txt 2>logs/sage-warnings3-stderr.txt

A summary of stderr output is

nbr_lines= 133
   83  Overriding __eq__ blocks inheritance of __hash__ in 3.x
   27  classic int division
   18  type inequality comparisons not supported in 3.x
    2  the cmp argument is not supported in 3.x
    1  execfile() not supported in 3.x; use exec()
    1  file.softspace not supported in 3.x
    1  tmpfile has been removed in 3.x; use the tempfile module

and of stdout

nbr_lines= 39630
19386  type inequality comparisons not supported in 3.x
 8779  Overriding __eq__ blocks inheritance of __hash__ in 3.x
 4039  CObject type is not supported in 3.x. Please use capsule objects instead.
 2139  reduce() not supported in 3.x; use functools.reduce()
 2017  classic int division
 2000  the cmp argument is not supported in 3.x
  789  comparing unequal types not supported in 3.x
  262  __getitem__ not supported for exception classes in 3.x; use args attribute
   88  sys.exc_clear() not supported in 3.x; use except clauses
   57  in 3.x, __getslice__ has been removed; use __getitem__
   14  file.softspace not supported in 3.x
   12  the commands module has been removed in Python 3.0; use the subprocess module instead
    7  execfile() not supported in 3.x; use exec()
    7  tmpfile has been removed in 3.x; use the tempfile module
    5  dict inequality comparisons not supported in 3.x
    3  
    2  Use the Permutations object instead.
    2  This class is replaced by Matrix_modn_dense_float/Matrix_modn_dense_double.
    2  to_permutation() is deprecated. Use instead reading_word_permutation()
    2  this class is deprecated. Use RibbonShapedTableau instead
    2  BaseException.message has been deprecated as of Python 2.6
    2  The 'new' module has been removed in Python 3.0; use the 'types' module instead.
    2  this is done automatically by the doctest framework
    2  derangements() is deprecated. Use Derangements instead.
    1  The output of Transducer.cartesian_product will change. Please use Transducer.intersection for the original output.
    1  Substitution using function-call syntax and unnamed arguments is deprecated and will be removed from a future release of Sage; you can use named arguments instead, like EXPR(x=..., y=...)
    1  This method is deprecated. Use triangulate() instead.
    1  In 3.x, reload() is renamed to imp.reload()
    1  dict.has_key() not supported in 3.x; use the in operator
    1  The output of
    1  Substitution using
    1  This method is deprecated. Use triangulate().simplicial_complex() instead.
    1  Variable of integration should be specified explicitly.
    1  OrderedAlphabet is deprecated; use Alphabet instead.

Unfortunately typical stdout printout looks like

**********************************************************************
File "src/sage/modular/modsym/space.py", line 929, in sage.modular.modsym.space.ModularSymbolsSpace._q_expansion_module
Failed example:
    ModularSymbols(11, 2, base_ring=GF(4,'a')).cuspidal_submodule()._q_expansion_module(prec=4, algorithm="hecke")
Expected:
    Vector space of degree 4 and dimension 1 over Finite Field in a of size 2^2
    Basis matrix:
    [0 1 0 1]
Got:
    doctest:127: DeprecationWarning: type inequality comparisons not supported in 3.x
    doctest:511: DeprecationWarning: classic int division
    doctest:250: DeprecationWarning: type inequality comparisons not supported in 3.x
    doctest:585: DeprecationWarning: classic int division
    doctest:286: DeprecationWarning: classic int division
    doctest:396: DeprecationWarning: classic int division
    doctest:6464: DeprecationWarning: classic int division
    Vector space of degree 4 and dimension 1 over Finite Field in a of size 2^2
    Basis matrix:
    [0 1 0 1]
**********************************************************************

and I don't know how to correlate line likes doctest:511 with the original line.

The line in stderr are more useful, but are few and far between

~/sage-6.2.beta8/local/lib/python2.7/site-packages/sage/groups/perm_gps/cubegroup.py:368: DeprecationWarning: classic int division
  2:  [1/2,3/2,5/2], #  ub

comment:3 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:4 Changed 6 years ago by aapitzsch

  • Dependencies set to #16371

Hi Wilfried, in #16371 I changed some / to //, maybe you can review it.

BTW: how you got the summary of stdout and stderr?

Changed 6 years ago by wluebbe

Small Python script to create the summary

comment:5 Changed 6 years ago by wluebbe

For the summary see the small attached Py script ;-)

I will look at your ticket.

comment:6 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:7 Changed 5 years ago by wluebbe

  • Dependencies changed from #16371 to #16371, #18578

I created the ticket #18578 to address classes which define the special function __div__() (and __truediv__()). See also the comments 5 and 7 in #16731.

comment:8 Changed 5 years ago by jdemeyer

One very small effort in the spirit of this ticket: #19535

comment:9 Changed 5 years ago by jdemeyer

Also #19536.

comment:10 in reply to: ↑ 1 Changed 5 years ago by tmonteil

Replying to wluebbe:

But currently I am stuck:

  • I don't know how to pass the option -Qwarnall to the Python interpreter when running the doctests

Could you try the following:

From a terminal, run sage -sh

From this shell, run:

export PYTHONIOENCODING="utf-8"
python -Qwarnall $(which sage-runtests) <options>

Where <options> is something like --all --long (depending on the options you are using for doctesting) ?

comment:11 Changed 4 years ago by jdemeyer

  • Component changed from distribution to python3

comment:12 Changed 4 years ago by chapoton

  • Description modified (diff)
  • Milestone changed from sage-6.4 to sage-7.3

comment:13 Changed 4 years ago by chapoton

  • Milestone changed from sage-7.3 to sage-7.4

one step done in #21127

comment:14 Changed 4 years ago by chapoton

  • Milestone changed from sage-7.4 to sage-7.6

comment:15 Changed 3 years ago by chapoton

another step in #22767

comment:16 Changed 3 years ago by chapoton

  • Milestone changed from sage-7.6 to sage-8.0

comment:17 Changed 3 years ago by chapoton

  • Milestone changed from sage-8.0 to sage-8.1

another step in #23731

comment:18 Changed 3 years ago by chapoton

another step in #24548

comment:19 Changed 3 years ago by embray

  • Description modified (diff)
  • Keywords division added; python3 removed
  • Owner changed from (none) to embray

comment:20 Changed 3 years ago by embray

  • Description modified (diff)

Some more status tracking for division-related work.

comment:21 Changed 3 years ago by embray

  • Description modified (diff)

Changed 2 years ago by chapoton

updated script

comment:22 Changed 16 months ago by chapoton

  • Cc embray jdemeyer fbissey tscrim jhpalmieri vklein added
  • Dependencies #16371, #18578 deleted
  • Milestone changed from sage-8.1 to sage-duplicate/invalid/wontfix
  • Status changed from new to needs_review

I think that we are now ready to close this one. Agreed ?

comment:23 Changed 16 months ago by jhpalmieri

  • Status changed from needs_review to positive_review

comment:24 Changed 16 months ago by chapoton

  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.