Opened 18 months ago

Closed 7 weeks ago

Last modified 7 weeks ago

#27754 closed enhancement (fixed)

Upgrade: Python 3.8.5

Reported by: slelievre Owned by:
Priority: blocker Milestone: sage-9.2
Component: packages: standard Keywords: days101
Cc: chapoton, embray, fbissey, jdemeyer, mkoeppe, slelievre, gh-timokau, isuruf, gh-kliem, tscrim, heluani, gh-sheerluck Merged in:
Authors: Jeroen Demeyer, John Palmieri, Matthias Koeppe, Antonio Rojas Reviewers: Matthias Koeppe, François Bissey
Report Upstream: Reported upstream. Developers acknowledge bug. Work issues:
Branch: f1cda12 (Commits) Commit:
Dependencies: #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365, #30140, #30351 Stopgaps:

Description (last modified by mkoeppe)

This is to upgrade to Python 3.8.x.

Tarball: see checksums.ini

Previous and next Python upgrades:

See also:

  • #30384 Adopt the “time window-based” policy for support of Python versions from NEP 29

Attachments (1)

sagelib-8.8.rc2.log (15.4 KB) - added by jhpalmieri 17 months ago.

Download all attachments as: .zip

Change History (222)

comment:1 Changed 18 months ago by slelievre

If people want to start testing, Python 3.8.0a3 is out:

comment:2 Changed 18 months ago by jdemeyer

I think it's a bit early for testing. I'd rather wait for the first beta, which is supposed to be released later this month.

comment:3 Changed 17 months ago by slelievre

Python 3.8.0b1 is out:

comment:4 Changed 17 months ago by jdemeyer

Note: this requires a Cython upgrade.

comment:5 Changed 17 months ago by slelievre

Cython upgrade needs review at #27886.

comment:6 Changed 17 months ago by slelievre

  • Dependencies set to #28012
  • Keywords days101 added

comment:7 Changed 17 months ago by jdemeyer

I'll have a look at this today.

comment:8 Changed 17 months ago by jdemeyer

  • Dependencies changed from #28012 to #27886

comment:9 Changed 17 months ago by jdemeyer

  • Authors set to Jeroen Demeyer
  • Description modified (diff)
  • Milestone changed from sage-wishlist to sage-8.9
  • Type changed from task to enhancement

comment:10 Changed 17 months ago by jdemeyer

  • Branch set to u/jdemeyer/upgrade__python_3_8_x

comment:11 Changed 17 months ago by git

  • Commit set to 46837b74871a41b8f79af5c9d477428b183dc2d7

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

46837b7Python 3.8.0

comment:12 Changed 17 months ago by jdemeyer

  • Dependencies #27886 deleted
  • Description modified (diff)

Actually, thanks to two patches which are likely going to be accepted upstream, it no longer depends on the Cython upgrade.

comment:13 Changed 17 months ago by jdemeyer

  • Report Upstream changed from N/A to Reported upstream. Developers acknowledge bug.

comment:14 Changed 17 months ago by jdemeyer

There is a mysterious build failure of matplotlib:

Processing /home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
  Created temporary directory: /tmp/pip-req-build-vq4okaxf
  Added file:///home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src to build tracker '/tmp/pip-req-tracker-xcv165b_'
  Running setup.py (path:/tmp/pip-req-build-vq4okaxf/setup.py) egg_info for package from file:///home/jdemeyer/sage-python3/local/var/tmp/sage/build/matplotlib-2.2.3.p0/src
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-vq4okaxf/setup.py", line 172, in <module>
        result = package.check()
      File "/tmp/pip-req-build-vq4okaxf/setupext.py", line 963, in check
        import numpy
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/__init__.py", line 16, in <module>
        from . import multiarray
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/multiarray.py", line 12, in <module>
        from . import overrides
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/core/overrides.py", line 8, in <module>
        from numpy.compat._inspect import getargspec
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/compat/__init__.py", line 14, in <module>
        from . import py3k
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/site-packages/numpy/compat/py3k.py", line 15, in <module>
        from pathlib import Path, PurePath
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/pathlib.py", line 391, in <module>
        class _NormalAccessor(_Accessor):
      File "/home/jdemeyer/sage-python3/local/lib/python3.8/pathlib.py", line 415, in _NormalAccessor
        link_to = os.link
    AttributeError: module 'os' has no attribute 'link'

But when running Python normally, os.link does exist:

jdemeyer@sage4:~/sage-python3$ ./sage --python3
Python 3.8.0b1 (default, Jun 19 2019, 16:54:26) 
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.link
<built-in function link>

I have no clue how this could happen.

comment:15 follow-ups: Changed 17 months ago by jhpalmieri

I tried this with OS X and ran into a problem with Pillow:

  File "setup.py", line 316, in build_extensions
    _add_directory(library_dirs, sysroot+"/usr/lib")
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

Python 3.7.3 does this:

Python 3.7.3 (default, Jun 17 2019, 15:38:22) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk'
>>>

The new Python does this:

Python 3.8.0b1 (default, Jun 19 2019, 11:09:54) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
>>>

comment:16 follow-up: Changed 17 months ago by jhpalmieri

I see the same problem with matplotlib. For laughs, I put a print(dir(os)) command right before the line link_to = os.link. The output is identical to what I get when I do import os; dir(os) from sage --python3, except it is missing link.

If I run make -k, I also see problems building the Sage library, by the way: lots of errors of the form

RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

comment:17 in reply to: ↑ 16 ; follow-up: Changed 17 months ago by jdemeyer

Replying to jhpalmieri:

I see the same problem with matplotlib. For laughs, I put a print(dir(os)) command right before the line link_to = os.link. The output is identical to what I get when I do import os; dir(os) from sage --python3, except it is missing link.

That makes no sense. Somebody is fooling with us...

Could you try also print(os)? Maybe there is a different os module shadowing the standard one or something...

Just in case, I checked that the problem unfortunately persists after make distclean.

And a Google search for this error doesn't find anything.

comment:18 in reply to: ↑ 17 Changed 17 months ago by jhpalmieri

I'm guessing the problem is coming from matplotlib's setup.py:

import os
try:
    del os.link
except AttributeError:
    pass

comment:19 follow-up: Changed 17 months ago by jhpalmieri

My question is why it worked with Python 3.7.3 but not 3.8.0.

comment:20 in reply to: ↑ 19 Changed 17 months ago by jdemeyer

Replying to jhpalmieri:

My question is why it worked with Python 3.7.3 but not 3.8.0.

A different implementation of pathlib: https://bugs.python.org/issue26978

Last edited 17 months ago by jdemeyer (previous) (diff)

comment:21 Changed 17 months ago by jdemeyer

  • Description modified (diff)

comment:22 Changed 17 months ago by jdemeyer

  • Dependencies set to #28023
  • Description modified (diff)

comment:23 Changed 17 months ago by git

  • Commit changed from 46837b74871a41b8f79af5c9d477428b183dc2d7 to cc78db22810ca605ed28235b45eeafe72418b269

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

296092ematplotlib 2.2.4
cc78db2Python 3.8.0

comment:24 Changed 17 months ago by jdemeyer

With matplotlib fixes, I'm getting these errors at runtime:

Exception ignored in: <sage.misc.weak_dict.WeakValueDictEraser object at 0x7f1bf60b5460>
Traceback (most recent call last):
  File "sage/misc/weak_dict.pyx", line 219, in sage.misc.weak_dict.WeakValueDictEraser.__call__ (build/cythonized/sage/misc/weak_dict.c:1968)
  File "sage/cpython/dict_del_by_value.pyx", line 375, in sage.cpython.dict_del_by_value.del_dictitem_by_exact_value (build/cythonized/sage/cpython/dict_del_by_value.c:2311)
  File "sage/cpython/dict_del_by_value.pyx", line 294, in sage.cpython.dict_del_by_value.ensure_allows_deletions (build/cythonized/sage/cpython/dict_del_by_value.c:2025)
AssertionError: 

comment:25 Changed 17 months ago by jdemeyer

  • Dependencies changed from #28023 to #28023, #28025

comment:26 Changed 17 months ago by jdemeyer

Also getargspec is completely broken, causing docbuild errors.

comment:27 follow-up: Changed 17 months ago by chapoton

see #27971 for getargspec ?

comment:28 in reply to: ↑ 27 Changed 17 months ago by jdemeyer

Replying to chapoton:

see #27971 for getargspec ?

I'll have to look at that. In any case, these seem to be new failures that did not exist on Python 3.7.

comment:29 in reply to: ↑ 15 ; follow-up: Changed 17 months ago by jhpalmieri

Replying to jhpalmieri:

I tried this with OS X and ran into a problem with Pillow: Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

This is actually added in a Sage patch. It's failing because of the removal of macos_no_include.patch, which was added in #27631. I'll have to see if that patch is necessary anymore. If not, we need to unpatch Pillow.

comment:30 Changed 17 months ago by jhpalmieri

I still gets lots of errors as in comment:16. The build process for sagelib seems to go into an infinite loop. I just hit ctrl-C, and there are about 2400 instances of

************************************************************************
Error building the Sage library
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/logs/pkgs/sagelib-8.8.rc2.log
Describe your computer, operating system, etc.

in the log file, all connected with the same error message.

comment:31 follow-up: Changed 17 months ago by jdemeyer

On which system are you seeing that? The build of the Sage library worked for me on Linux.

comment:32 in reply to: ↑ 31 Changed 17 months ago by jhpalmieri

Replying to jdemeyer:

On which system are you seeing that? The build of the Sage library worked for me on Linux.

OS X, both 10.13 and 10.14.

comment:33 in reply to: ↑ 29 Changed 17 months ago by jhpalmieri

Replying to jhpalmieri:

Replying to jhpalmieri:

I tried this with OS X and ran into a problem with Pillow: Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

This is actually added in a Sage patch. It's failing because of the removal of macos_no_include.patch, which was added in #27631. I'll have to see if that patch is necessary anymore. If not, we need to unpatch Pillow.

On an OS X 10.14 machine, one affected by #27631 (no /usr/include), pip builds just fine, so I think Python and pip are finding zlib, but Pillow does not.

If I keep the Pillow patch, it fails to build at all, with the error mentioned above. If I delete the Pillow patch and build, I see in its log file "The headers and library files could not be found for zlib".

Last edited 17 months ago by jhpalmieri (previous) (diff)

Changed 17 months ago by jhpalmieri

comment:34 Changed 17 months ago by jhpalmieri

Here is the start of the log file. I've deleted more than 2000 repeated tracebacks.

comment:35 Changed 16 months ago by jhpalmieri

I think the problem is this (from the 3.8 changelog):

on macOS, the spawn start method is now used by default in multiprocessing

The documentation for multiprocessing then points to a bug report which discusses our old friend OBJC_DISABLE_INITIALIZE_FORK_SAFETY (#25921). So if we get multiprocessing to work properly on OS X, maybe we can remove the line

    export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

from sage-env.

In the meantime, I don't know how to get it working properly; using fork instead of spawn in src/setup.py is the only way I've found to proceed so far. I don't understand the distinctions in the multiprocessing documentation between fork and spawn to know what should be set differently to use spawn correctly.

comment:36 Changed 16 months ago by jhpalmieri

Using fork instead of spawn lets me build the Sage library:

  • src/setup.py

    diff --git a/src/setup.py b/src/setup.py
    index 42ba3a63da..5e2d65f9cc 100755
    a b def execute_list_of_commands_in_parallel(command_list, nthreads): 
    493493        progress = progress_fmt.format(i+1, N)
    494494        command_list[i] = command_list[i] + (progress,)
    495495
    496     from multiprocessing import Pool
     496    import multiprocessing as mp
     497    mp.set_start_method('fork')
    497498    # map_async handles KeyboardInterrupt correctly if an argument is
    498499    # given to get().  Plain map() and apply_async() do not work
    499500    # correctly, see Trac #16113.
    500     pool = Pool(nthreads)
     501    pool = mp.Pool(nthreads)
    501502    result = pool.map_async(apply_func_progress, command_list, 1).get(99999)
    502503    pool.close()
    503504    pool.join()

I made a similar change to allow docbuilding to work. Now I get failures with docbuilding. First, lots of warnings:

..../local/lib/python3.8/site-packages/matplotlib/backends/backend_agg.py:531: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats

Then an error:

Traceback (most recent call last):
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
    main()
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 1731, in main
    builder()
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 352, in _wrapper
    getattr(get_builder(document), 'inventory')(*args, **kwds)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 548, in _wrapper
    build_many(build_ref_doc, L)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/site-packages/sage_setup/docbuild/__init__.py", line 289, in build_many
    ret = x.get(99999)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2/local/lib/python3.8/multiprocessing/pool.py", line 768, in get
    raise self._value
OSError: WARNING: error while formatting arguments for sage.calculus.riemann.Riemann_Map.get_szego: bad operand type for unary -: 'NoneType'

comment:37 Changed 16 months ago by jhpalmieri

And of course I can't run any doctests because they use multiprocessing, and I haven't figured out how to tell doctesting (forker.py in particular) to use fork instead of spawn. Sage starts and some basic things work. The Jupyter notebook is giving me problems.

Last edited 16 months ago by jhpalmieri (previous) (diff)

comment:38 Changed 16 months ago by jdemeyer

I also didn't manage to build the docs because the same error. I know it's a problem with sage_getargspec but I haven't checked further.

comment:39 Changed 16 months ago by jhpalmieri

I don't have a good solution for the problem with Pillow (comment:33). I have been deleting the patch and force installing zlib, which works for now.

comment:40 Changed 16 months ago by jhpalmieri

For the record: for the spawn vs. fork issue with OS X, I have had to make changes in three files: src/setup.py, src/sage_setup/docbuild/__init__.py, and Cython/Build/Dependencies.py. The changes are all pretty much the same; for example in docbuild/__init__.py:

  • src/sage_setup/docbuild/__init__.py

    diff --git a/src/sage_setup/docbuild/__init__.py b/src/sage_setup/docbuild/__init__.py
    index 0f2700168a..6d1ff4e377 100644
    a b if not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3): 
    279279        # serialize this by taking the pool (and thus the call to fork()) out
    280280        # completely: The call to Sphinx leaks memory, so we need to build each
    281281        # document in its own process to control the RAM usage.
    282         from multiprocessing import Pool
    283         pool = Pool(NUM_THREADS, maxtasksperchild=1)
     282        import multiprocessing
     283        ctx = multiprocessing.get_context('fork')
     284        pool = ctx.Pool(NUM_THREADS, maxtasksperchild=1)
    284285        # map_async handles KeyboardInterrupt correctly. Plain map and
    285286        # apply_async does not, so don't use it.
    286287        x = pool.map_async(target, args, 1)

comment:41 follow-up: Changed 16 months ago by jhpalmieri

I am also seeing the error "ImportError?: cannot import name 'clock' from 'time' (unknown location)". I don't think this is coming from the Sage library, but rather from other packages, and in particular brial and sympy may need to be changed.

$ grep -R "from time import clock" local/lib/python3.8/
local/lib/python3.8//site-packages/sympy/plotting/pygletplot/plot_window.py:from time import clock
local/lib/python3.8//site-packages/brial/interpolate.py:from time import clock
local/lib/python3.8//site-packages/future/backports/test/pystone.py:from time import clock
local/lib/python3.8//site-packages/zmq/backend/cffi/_poll.py:    from time import clock as monotonic
local/lib/python3.8//site-packages/sage/coding/information_set_decoder.py:            from time import clock as process_time

Changing the import to something like

try:
    from time import perf_counter as clock # or process_time instead of perf_counter?
except ImportError:
    from time import clock

might work.

comment:42 in reply to: ↑ 41 Changed 16 months ago by chapoton

Replying to jhpalmieri:

I am also seeing the error "ImportError?: cannot import name 'clock' from 'time' (unknown location)". I don't think this is coming from the Sage library, but rather from other packages, and in particular brial and sympy may need to be changed.

$ grep -R "from time import clock" local/lib/python3.8/
local/lib/python3.8//site-packages/sympy/plotting/pygletplot/plot_window.py:from time import clock
local/lib/python3.8//site-packages/brial/interpolate.py:from time import clock
local/lib/python3.8//site-packages/future/backports/test/pystone.py:from time import clock
local/lib/python3.8//site-packages/zmq/backend/cffi/_poll.py:    from time import clock as monotonic
local/lib/python3.8//site-packages/sage/coding/information_set_decoder.py:            from time import clock as process_time

Changing the import to something like

try:
    from time import perf_counter as clock # or process_time instead of perf_counter?
except ImportError:
    from time import clock

might work.

Last edited 16 months ago by chapoton (previous) (diff)

comment:43 Changed 16 months ago by jdemeyer

  • Dependencies changed from #28023, #28025 to #28023, #28025, #28116
  • Description modified (diff)

comment:44 Changed 14 months ago by jdemeyer

  • Dependencies changed from #28023, #28025, #28116 to #28025
  • Description modified (diff)

comment:45 Changed 14 months ago by jdemeyer

Python 3.8 is getting closer, it would be great if somebody could review the dependency #28025.

comment:46 Changed 14 months ago by jdemeyer

  • Dependencies changed from #28025 to #28025, #28428

comment:47 Changed 13 months ago by slelievre

  • Description modified (diff)

Python 3.8.0 is out. Added tarball link to ticket description.

comment:48 Changed 12 months ago by chapoton

  • Branch changed from u/jdemeyer/upgrade__python_3_8_x to public/python3.8
  • Commit changed from cc78db22810ca605ed28235b45eeafe72418b269 to 193e9f49517a95e483af785d9879948468e9ed98
  • Dependencies #28025, #28428 deleted
  • Milestone changed from sage-8.9 to sage-9.0

New commits:

193e9f4Python 3.8.0

comment:49 Changed 12 months ago by chapoton

apparently some patches do no longer apply..

comment:50 Changed 12 months ago by jhpalmieri

It looks to me as though the two new patches were actually merged into the release, and so they are no longer needed. I'm not positive, though.

comment:51 Changed 12 months ago by jhpalmieri

On OS X, the Pillow problem is still there. I tried upgrading to the most recent version of Pillow, but it did not help. I'm going to force installation of zlib and proceed from there.

comment:52 Changed 12 months ago by jhpalmieri

There is still a problem with a from time import clock statement. I don't know which one is causing the problem, but I'm guessing either brial or sympy. Neither package has fixed it in a release, as far as I can tell, but I think it's been fixed upstream in both.

comment:53 Changed 12 months ago by jhpalmieri

I am also, not surprisingly, having the same multiprocessing problems on OS X as before. I can get the Sage library and the documentation to build, but I don't know how to get doctesting to work.

comment:54 Changed 12 months ago by arojas

FWIW after updating to python 3.8 on Arch I'm getting test failures on 223 modules, so things don't look good. Most of them seem to stem from the same 3-4 issues though.

comment:55 follow-ups: Changed 11 months ago by arojas

After taking a closer look, besides some string changes, most of the failures fall into one of the following categories:

(1) Hash changes - all hashes have changed in 3.8. This requires updating many tests, making them conditional on the python version. Not sure if it makes sense to fork so many tests at this point rather than just drop python2 support or at least not run those tests.

(2) Sorting changes - many lists are now output in a different order. Same comment as in (1) about the solution.

(3) "DeprecationWarning?: In future, it will be an error for 'np.bool_' scalars to be interpreted as an index" from numpy. These are fixed with https://github.com/matplotlib/matplotlib/commit/4651a125

(4) "DeprecationWarning?: PY_SSIZE_T_CLEAN will be required for '#' formats" - this happens mostly when pickling/unpickling functions. No idea where the actual code that throws the warning is.

(5) "DeprecationWarning?: an integer is required (got type sage.rings.real_mpfr.RealLiteral?). Implicit conversion to integers using int is deprecated, and may be removed in a future version of Python." when calling time.sleep with a Sage (sage.rings.real_mpfr.RealLiteral?) float. Coercing to float first fixes the issue.

(6) "SyntaxWarning?: "is" with a literal. Did you mean "=="?" when comparing strings with 'is'

(7) Use of the removed cgi.escape in sage/interacts/debugger.py. Should be changed to html.escape.

(8) The signature of code() (used in sage/misc/fpickle.pyx) has changed and now requires a 'posonlyargcount' argument.

(9) Mismatch after pickling/unpickling for some objects:

sage: A = Poset()._hasse_diagram.antichains()
sage: A == loads(dumps(A))
False

For packagers that don't care about python2 compatibility (or 32 bit hashes) a WIP patch that fixes or works around most of these issues is at https://aur.archlinux.org/cgit/aur.git/tree/sagemath-python-3.8.patch?h=sagemath-git. A backport of https://github.com/BRiAl/BRiAl/commit/74d86170 is also needed.

comment:56 Changed 11 months ago by gh-timokau

  • Cc gh-timokau added

comment:58 follow-up: Changed 11 months ago by jhpalmieri

It would be great if someone with access to OS X could look at this. I don't know how to solve all of the problems mentioned above. In particular:

  • multiprocessing on OS X has a different default behavior than on linux (comment:35). As a result, the Sage library doesn't build and the docs don't build without some modification, and I don't know how to get doctesting to work.
  • Pillow does not find the system zlib. Is there a better solution than force-installing Sage's zlib?
Last edited 11 months ago by jhpalmieri (previous) (diff)

comment:59 Changed 10 months ago by slelievre

I have access to macOS but I would need to be told what to try.

comment:60 follow-up: Changed 10 months ago by slelievre

Reports of issues with SageMath and Python 3.8:

comment:61 in reply to: ↑ 60 Changed 10 months ago by arojas

Replying to slelievre:

Reports of issues with SageMath and Python 3.8:

None of these is a Sage issue: the first one is caused by the user mixing up distro and pip packages, the second one is a ntl soversion mismatch in Manjaro. The errors that still need to be figured out (besides the macOS problems) are (4), (5) and (9) in comment:55 above.

Last edited 10 months ago by arojas (previous) (diff)

comment:62 Changed 10 months ago by embray

  • Milestone changed from sage-9.0 to sage-9.2

Sage 9.0 is out. I think this should wait until after 9.1 to give Sage with Python 3.7 a little more time to breathe, but of course this is debatable.

Last edited 10 months ago by embray (previous) (diff)

comment:63 in reply to: ↑ 58 Changed 10 months ago by mkoeppe

Replying to jhpalmieri:

  • Pillow does not find the system zlib. Is there a better solution than force-installing Sage's zlib?

I have created #29019 for the problems with Pillow because they also affect #27824.

comment:64 in reply to: ↑ 15 Changed 10 months ago by mkoeppe

Replying to jhpalmieri:

I tried this with OS X and ran into a problem with Pillow:

  File "setup.py", line 316, in build_extensions
    _add_directory(library_dirs, sysroot+"/usr/lib")
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Here is the problem code from Pillow's setup.py:

sysroot = sysconfig.get_config_var('Py_MACOS_SYSROOT')
_add_directory(library_dirs, sysroot+"/usr/lib")

Python 3.7.3 does this:

Python 3.7.3 (default, Jun 17 2019, 15:38:22) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
'/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk'
>>>

The new Python does this:

Python 3.8.0b1 (default, Jun 19 2019, 11:09:54) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var('Py_MACOS_SYSROOT')
>>>

As far as I can see, this config variable was invented in #27631 and is not available on any other Python than our patched Python.

comment:65 Changed 10 months ago by dimpase

indeed, Py_MACOS_SYSROOT comes from #27631. Note that #27631 followed a cpython patch we proposed on https://bugs.python.org/issue36231 which was rejected in favour of https://github.com/python/cpython/commit/c421c66a58a6caae30f0679d7e61411418e67cec

and so in 3.7.x with x>3 and in 3.y with y>7 the problem apparently is fixed, but I don't quite know how. I don't have any access to a modern MacOS box with Xcode without sane /usr/incude etc, so it's hard to experiment for me (I guess the patch for pillow in #27631 needs to be replaced with whatever hack they came up with for MacOS.

comment:66 follow-up: Changed 10 months ago by jhpalmieri

See comment:33 — I could not build Pillow (after removing Sage's patch) with Python 3.8. So I don't know if I believe that the problem has been fixed.

comment:67 in reply to: ↑ 66 ; follow-up: Changed 10 months ago by mkoeppe

Replying to jhpalmieri:

See comment:33 — I could not build Pillow (after removing Sage's patch) with Python 3.8. So I don't know if I believe that the problem has been fixed.

Please try again with the latest commit in #29019.

comment:68 Changed 10 months ago by git

  • Commit changed from 193e9f49517a95e483af785d9879948468e9ed98 to 9cf6f0ebf83c57384161fdc1594bc8156e1c49c3

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

9c6324bPython 3.8.0
9cf6f0etrac 27754: remove patches that have been merged upstream

comment:69 in reply to: ↑ 67 Changed 10 months ago by jhpalmieri

Replying to mkoeppe:

Replying to jhpalmieri:

See comment:33 — I could not build Pillow (after removing Sage's patch) with Python 3.8. So I don't know if I believe that the problem has been fixed.

Please try again with the latest commit in #29019.

That works for me. With this branch plus #29019, the build succeeds until I get to the Sage library, at which point the changes to multiprocessing start causing problems (as in comment:16, comment:35, comment:36).

comment:70 Changed 10 months ago by jhpalmieri

See https://github.com/cython/cython/issues/3262 and https://github.com/conda-forge/cython-feedstock/pull/59 (?) for multiprocessing issues with Cython + Python 3.8 + OS X. One proposed solution there is to disable parallel processing on OS X (or more generally if multiprocessing.get_start_method() == 'spawn'). Do we want to do that, or are we willing to use a patch like the proposal in comment:40, applied to Cython/Build/Dependencies.py?

comment:71 Changed 10 months ago by jhpalmieri

And by the way, even with the patches as in comment:40, I still can't get the documentation to build. It ends with

[dochtml]     ForkingPickler(file, protocol).dump(obj)
[dochtml] AttributeError: Can't pickle local object 'build_many.<locals>.run_worker'

comment:72 in reply to: ↑ 55 ; follow-up: Changed 7 months ago by gh-timokau

Replying to arojas: [...]

For packagers that don't care about python2 compatibility (or 32 bit hashes) a WIP patch that fixes or works around most of these issues is at https://aur.archlinux.org/cgit/aur.git/tree/sagemath-python-3.8.patch?h=sagemath-git. A backport of https://github.com/BRiAl/BRiAl/commit/74d86170 is also needed.

@arojas, I'm trying to update sage on nix to 9.0 with python 3.8. I'm using your 3.8 patch in the 9.0 version [1]. I'm still seeing some test failures caused by the python update though:

sage -t --long src/sage/categories/homset.py  # 3 doctests failed
sage -t --long src/sage/combinat/cluster_algebra_quiver/quiver_mutation_type.py  # 1 doctest failed
sage -t --long src/sage/combinat/finite_state_machine.py  # 1 doctest failed
sage -t --long src/sage/combinat/posets/hasse_diagram.py  # 2 doctests failed
sage -t --long src/sage/groups/cubic_braid.py  # 1 doctest failed
sage -t --long src/sage/homology/simplicial_complex_morphism.py  # 1 doctest failed
sage -t --long src/sage/structure/sequence.py  # 2 doctests failed

Here[2] is the full log. You said that your patch fixes "most" failures, so is that to be expected? Do you have the same failures on arch?

[1] https://aur.archlinux.org/cgit/aur.git/plain/sagemath-python-3.8.patch?h=sagemath-git&id=138b3db82d8d13f9c66bf8d7515aa37671710e0a [2] https://gist.github.com/timokau/13c97bf8f45162f3d4155999fc5bec13

comment:73 in reply to: ↑ 72 Changed 7 months ago by arojas

Replying to gh-timokau:

Here[2] is the full log. You said that your patch fixes "most" failures, so is that to be expected? Do you have the same failures on arch?

Yes, I have the same failures (the one in cubic_braid no longer happens in the latest beta). They are mostly related to pickling, which I don't understand.

comment:74 Changed 7 months ago by gh-timokau

Thanks!

I had a look at the python 3.8 changelog, the most obvious change seemed to be the change of the default pickle protocol (3 -> 4). But I have patched python to revert that change and that did not fix the issue.

I'll look into git-bisect'ing the cpython update, though that is of course not exactly trivial. Maybe it will help.

comment:75 Changed 7 months ago by gh-jaliste

there is a change of API in the ast, so this breaks old versions of ipython with python 3.8. In particular, it breaks ipython 5.8 so you cannot run sage under jupyter in python 3.8 so far.

comment:76 Changed 7 months ago by mkoeppe

  • Dependencies set to #28197

I've added as a dependency: #28197 - upgrade to ipython 7

comment:77 in reply to: ↑ 55 Changed 7 months ago by thansen

Replying to arojas:

For packagers that don't care about python2 compatibility (or 32 bit hashes) a WIP patch that fixes or works around most of these issues is at https://aur.archlinux.org/cgit/aur.git/tree/sagemath-python-3.8.patch?h=sagemath-git.

Thanks for the patch. A version of this patch including the 32 bit hashes can now be found at https://salsa.debian.org/science-team/sagemath/-/blob/master/debian/patches/u0-version-python-3.8.patch.

Unfortunately many of the ordering issues addressed in this patch also depend on 32/64 bit now. See 32-bit build log at https://buildd.debian.org/status/fetch.php?pkg=sagemath&arch=i386&ver=9.0-3&stamp=1586083972&raw=0.

comment:78 Changed 5 months ago by mkoeppe

  • Dependencies changed from #28197 to #28197, #29042

comment:79 Changed 5 months ago by mkoeppe

  • Cc isuruf added

comment:80 Changed 4 months ago by mkoeppe

Can we get this ticket unstuck please?

comment:81 Changed 4 months ago by jhpalmieri

I am seeing negative progress: I can't even get the Sage library to build on OS X anymore: I just see errors as in comment:16. I assume the problem is parallel building with spawn instead of fork, but I can't figure out where to change things to get it to work.

For the record, I tried this with Python 3.8.4, not 3.8.0. I'll try with 3.8.0, but I don't expect any difference. I tried this but it didn't help:

  • src/sage_setup/run_parallel.py

    diff --git a/src/sage_setup/run_parallel.py b/src/sage_setup/run_parallel.py
    index 7a9b787b4e..a4cf73801d 100644
    a b def execute_list_of_commands_in_parallel(command_list, nthreads): 
    7373        progress = progress_fmt.format(i+1, N)
    7474        command_list[i] = command_list[i] + (progress,)
    7575
    76     from multiprocessing import Pool
     76    import multiprocessing
     77    ctx = multiprocessing.get_context('fork')
    7778    # map_async handles KeyboardInterrupt correctly if an argument is
    7879    # given to get().  Plain map() and apply_async() do not work
    7980    # correctly, see Trac #16113.
    80     pool = Pool(nthreads)
     81    pool = ctx.Pool(nthreads, maxtasksperchild=1)
    8182    result = pool.map_async(apply_func_progress, command_list, 1).get(99999)
    8283    pool.close()
    8384    pool.join()

comment:82 Changed 4 months ago by git

  • Commit changed from 9cf6f0ebf83c57384161fdc1594bc8156e1c49c3 to 9688a9c6064603725c6aca4278f7098b758743c9

Branch pushed to git repo; I updated commit sha1. New commits:

c119cf2Merge tag '9.2.beta5' into t/27754/public/python3.8
9688a9cbuild/pkgs/python3: Update to 3.8.4

comment:83 Changed 4 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Upgrade: Python 3.8.x to Upgrade: Python 3.8.4

comment:84 Changed 4 months ago by mkoeppe

I'll try to fix / work around the build problem

comment:86 Changed 4 months ago by git

  • Commit changed from 9688a9c6064603725c6aca4278f7098b758743c9 to c44c448acfb4152e981d91ec02b4eb6ae34c57a7

Branch pushed to git repo; I updated commit sha1. New commits:

c24d109build/pkgs/cython: Updatt to 0.29.21
beb8c8aMerge branch 't/29861/upgrade_cython_to_0_29_20' into t/27754/public/python3.8
c44c448src/setup.py: Work around a Cython problem with multiprocessing in Python 3.8.x on macOS

comment:87 Changed 4 months ago by mkoeppe

Not sure if the Cython update (from #29861) was necessary. But 0.29.21 does have a change specifically for 3.8.

comment:88 Changed 4 months ago by mkoeppe

sagelib builds now on my Mac. Haven't tested anything else yet.

comment:89 Changed 4 months ago by jhpalmieri

Right, thanks, I forgot to deal with Cython. I can now build sagelib. Also the docs, with the following changes:

  • src/sage_setup/docbuild/utils.py

    diff --git a/src/sage_setup/docbuild/utils.py b/src/sage_setup/docbuild/utils.py
    index 272f7e2d0f..2691339ea6 100644
    a b def build_many(target, args, processes=None): 
    112112        ...
    113113        WorkerDiedException: worker for 4 died with non-zero exit code -9
    114114    """
    115     from multiprocessing import Process, Queue, cpu_count
     115    import multiprocessing
     116    ctx = multiprocessing.get_context('fork')
     117    Process = ctx.Process
     118    Queue = ctx.Queue
    116119    from six.moves.queue import Empty
    117120
    118121    if processes is None:
    119         processes = cpu_count()
     122        processes = multiprocessing.cpu_count()
    120123
    121124    workers = [None] * processes
    122125    tasks = enumerate(args)

Doctesting doesn't work, though; first, it failed completely, and then I tried to patch a few files in src/sage/doctest, but I was not successful. (I got some tests to run, but there were doctest failures, and then the process just quit.)

comment:90 Changed 4 months ago by mkoeppe

According to https://docs.python.org/3/library/multiprocessing.html#multiprocessing.get_context, get_context is available for python >= 3.4. We support python >= 3.6, so I would guess it is safe to make this change to utils.py. Could you push it to the branch please?

comment:91 Changed 3 months ago by git

  • Commit changed from c44c448acfb4152e981d91ec02b4eb6ae34c57a7 to 6a8c543c70ee90b5ad332ace5aebf5e498fb1444

Branch pushed to git repo; I updated commit sha1. New commits:

6a8c543trac 27754: docbuilding on OS X

comment:92 Changed 3 months ago by jhpalmieri

This builds sagelib plus the documentation for me on, either using the system Python or Python 3.8.4.

comment:93 Changed 3 months ago by jhpalmieri

  • Dependencies changed from #28197, #29042 to #28197, #29042, #29861

comment:94 Changed 3 months ago by mkoeppe

Thanks. Do we know if the problem with doctesting is specific to macOS?

comment:95 Changed 3 months ago by jhpalmieri

With this branch, everything builds on an Ubuntu virtual machine. I didn't run doctests.

I also didn't know about the

multiprocessing.set_start_method('fork', force=True)

command. With this change, doctests on OS X run, although there are a lot of failures:

  • src/sage/doctest/external.py

    diff --git a/src/sage/doctest/external.py b/src/sage/doctest/external.py
    index b0db07b47a..af49ca29e5 100644
    a b AUTHORS: 
    2525#                  http://www.gnu.org/licenses/
    2626#*****************************************************************************
    2727
    28 from multiprocessing import Array
     28import multiprocessing
     29import os
     30if os.uname().sysname == 'Darwin':
     31    multiprocessing.set_start_method('fork', force=True)
     32Array = multiprocessing.Array
    2933
    3034import urllib.error
    3135from urllib.request import Request, urlopen
  • src/sage/doctest/forker.py

    diff --git a/src/sage/doctest/forker.py b/src/sage/doctest/forker.py
    index 75559667d1..8f5f2bff2b 100644
    a b from sage.repl.user_globals import set_globals 
    6363from sage.cpython.atexit import restore_atexit
    6464from sage.cpython.string import bytes_to_str, str_to_bytes
    6565
     66if os.uname().sysname == 'Darwin':
     67    multiprocessing.set_start_method('fork', force=True)
    6668
    6769# All doctests run as if the following future imports are present
    6870import __future__

comment:96 Changed 3 months ago by mkoeppe

Could you push it on the branch?

comment:97 Changed 3 months ago by mkoeppe

Without the changes from comment 95, I get the following error:

$ ./sage -t src/sage/geometry/polyhedron/base.py
too many failed tests, not using stored timings
Running doctests with ID 2020-07-15-19-42-59-9c236f91.
Git branch: big-update
Using --optional=build,cryptominisat,dochtml,sage
Doctesting 1 file.
----------------------------------------------------------------------
Doctests interrupted: 0/1 files tested
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds
Traceback (most recent call last):
  File "/Users/mkoeppe/s/sage/sage-rebasing/src/bin/sage-runtests", line 177, in <module>
    err = DC.run()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/control.py", line 1207, in run
    self.run_doctests()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/control.py", line 908, in run_doctests
    self.dispatcher.dispatch()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 2039, in dispatch
    self.parallel_dispatch()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 1934, in parallel_dispatch
    w.start()  # This might take some time
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 2206, in start
    super(DocTestWorker, self).start()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'DocTestDispatcher.parallel_dispatch.<locals>.sel_exit'

Is this the same error that you were getting?

comment:98 Changed 3 months ago by git

  • Commit changed from 6a8c543c70ee90b5ad332ace5aebf5e498fb1444 to 5a5fa82344129e34f03b62ec2c3bb0a72de0d808

Branch pushed to git repo; I updated commit sha1. New commits:

5a5fa82trac 27754: on OS X, use 'fork' with multiprocessing when doctesting

comment:99 Changed 3 months ago by jhpalmieri

Yes, that's the error I was getting. Now I'm getting the same doctest failures on OS X as on the Ubuntu virtual machine (approximately: I think I get 113 files with failures on OS X, 114 with failures on Ubuntu).

comment:100 Changed 3 months ago by mkoeppe

I tried make ptest, and it ended with:

sage -t src/sage/calculus/riemann.pyx  # 18 doctests failed
sage -t src/sage/functions/log.py  # 2 doctests failed
sage -t src/sage/graphs/graph_plot.py  # 4 doctests failed
Doctests interrupted: 344/4127 files tested
  File "/Users/mkoeppe/s/sage/sage-rebasing/src/bin/sage-runtests", line 177, in <module>
    err = DC.run()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/control.py", line 1207, in run
    self.run_doctests()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/control.py", line 908, in run_doctests
    self.dispatcher.dispatch()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 2044, in dispatch
    self.parallel_dispatch()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 1939, in parallel_dispatch
    w.start()  # This might take some time
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 2211, in start
    super(DocTestWorker, self).start()
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/mkoeppe/s/sage/sage-rebasing/local/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
    child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files
make: *** [ptest] Error 1

comment:101 Changed 3 months ago by jhpalmieri

make ptestlong completed for me (with make -j6). I see several types of failures:

  • sorting issues:
    File "src/sage/dynamics/arithmetic_dynamics/projective_ds.py", line 3148, in sage.dynamics.arithmetic_dynamics.projective_ds.?.automorphism_group
    Failed example:
        D6.automorphism_group()
    Expected:
        [
        [1 0]  [0 w]  [0 1]  [w 0]  [-w - 1      0]  [     0 -w - 1]
        [0 1], [1 0], [1 0], [0 1], [     0      1], [     1      0]
        ]
    Got:
        [
        [1 0]  [0 1]  [     0 -w - 1]  [w 0]  [-w - 1      0]  [0 w]
        [0 1], [1 0], [     1      0], [0 1], [     0      1], [1 0]
        ]
    
  • hashes have changed:
    File "src/sage/combinat/similarity_class_type.py", line 411, in sage.combinat.similarity_class_type.?.__hash__
    Failed example:
        hash(PT1)
    Expected:
        5050909583595644741 
    Got:
        -925386691174542829
    
  • pickling?
    File "src/sage/manifolds/differentiable/affine_connection.py", line 1785, in sage.manifolds.differentiable.affine_connection.AffineConnection.riemann
    Failed example:
        for i in M.irange():
            for j in M.irange():
                for k in M.irange():
                    nab.add_coef(eV)[i,j,k] = nab.coef(eVW)[i,j,k,c_uvW].expr()
    Exception raised:
        multiprocessing.pool.RemoteTraceback: 
        """
        Traceback (most recent call last):
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/multiprocessing/pool.py", line 125, in worker
            result = (True, func(*args, **kwds))
          File "sage/misc/fpickle.pyx", line 99, in sage.misc.fpickle.call_pickled_function (build/cythonized/sage/misc/fpickle.c:2279)
            f = eval("unpickle_function(fp)", sage.all.__dict__, {'fp':fp})
          File "<string>", line 1, in <module>
          File "sage/misc/fpickle.pyx", line 90, in sage.misc.fpickle.unpickle_function (build/cythonized/sage/misc/fpickle.c:1975)
            recovered = pickle.loads(pickled)
          File "sage/misc/fpickle.pyx", line 26, in sage.misc.fpickle.code_ctor (build/cythonized/sage/misc/fpickle.c:1536)
            return types.CodeType(*args)
        TypeError: code() takes at least 14 arguments (13 given)
        """
    
        The above exception was the direct cause of the following exception:
    
        Traceback (most recent call last):
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 709, in _run
            self.compile_and_execute(example, compiler, test.globs)
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
            exec(compiled, globs)
          File "<doctest sage.manifolds.differentiable.affine_connection.AffineConnection.riemann[30]>", line 4, in <module>
            nab.add_coef(eV)[i,j,k] = nab.coef(eVW)[i,j,k,c_uvW].expr()
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/site-packages/sage/manifolds/differentiable/affine_connection.py", line 663, in coef
            nab_evi = self(ev[i])
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/site-packages/sage/manifolds/differentiable/affine_connection.py", line 1386, in __call__
            return self._derive_paral(tensor_r)
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/site-packages/sage/manifolds/differentiable/affine_connection.py", line 1508, in _derive_paral
            for ii,val in make_CovDerivative(listParalInput):
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/site-packages/sage/parallel/multiprocessing_sage.py", line 77, in parallel_iter
            for res in result:
          File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta5/local/lib/python3.8/multiprocessing/pool.py", line 868, in next
            raise value
        TypeError: code() takes at least 14 arguments (13 given)
    
  • a deprecation warning:
    File "src/sage/plot/plot3d/base.pyx", line 1560, in sage.plot.plot3d.base.Graphics3d.save_image
    Failed example:
        G.save_image(gif)
    Expected nothing
    Got:
        doctest:warning
        ...
        DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
    
  • another one:
    File "src/sage/doctest/forker.py", line 2351, in sage.doctest.forker.DocTestWorker.kill
    Failed example:
        time.sleep(0.2)  # Worker doesn't die
    Expected nothing
    Got:
        doctest:warning
        ...
        DeprecationWarning: an integer is required (got type sage.rings.real_mpfr.RealLiteral).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
    

I haven't gone through all of the failures yet, nor have I pored over the Python 3.8 changelog to try to figure any of these out. The first few types are easy to fix, I hope.

comment:102 Changed 3 months ago by mkoeppe

comment 77 above already mentions the hashing/ordering issues. I haven't looked at these patches

comment:103 Changed 3 months ago by mkoeppe

If doctests need patching because of sorting order changes, I hope we can do better than what I see in https://salsa.debian.org/science-team/sagemath/-/blob/master/debian/patches/u0-version-python-3.8.patch

Many of such tests should in my opinion be fixed by using explicitly sorted when comparing lists whose order has no mathematical meaning

comment:104 Changed 3 months ago by mkoeppe

In this one from the Debian patch, I'm wondering whether printing of Set in doctests should be modified similar to what was done in #29042 for dict

  • sage/src/sage/schemes/elliptic_curves/ell_point.py

    a b  
    822822            sage: all(T.is_divisible_by(3) for T in tor)
    823823            True
    824824            sage: Set([T for T in tor if T.is_divisible_by(2)])
    825             {(0 : 1 : 0), (1 : 0 : 1)}
     825            {(1 : 0 : 1), (0 : 1 : 0)}
    826826            sage: Set([2*T for T in tor])
    827             {(0 : 1 : 0), (1 : 0 : 1)}
     827            {(1 : 0 : 1), (0 : 1 : 0)}
    828828
Last edited 3 months ago by mkoeppe (previous) (diff)

comment:105 Changed 3 months ago by mkoeppe

  • Cc gh-kliem added

comment:106 Changed 3 months ago by mkoeppe

For these ones from the Debian patch I have created #30162:

--- a/sage/src/sage/symbolic/callable.py
+++ b/sage/src/sage/symbolic/callable.py
@@ -38,27 +38,27 @@
     sage: f(1)=2
     Traceback (most recent call last):
     ...
-    SyntaxError: can't assign to function call
+    SyntaxError: cannot assign to function call

comment:107 Changed 3 months ago by mkoeppe

  • Cc tscrim added

For this one from the Debian patch, I also wonder whether some methods should be changed so that they return generators rather than lists - for better py3 style.

--- a/sage/src/sage/categories/coxeter_groups.py
+++ b/sage/src/sage/categories/coxeter_groups.py
@@ -283,7 +283,7 @@
                 [[0, 1, 2, 1], [0, 2, 1, 2], [2, 0, 1, 2]]
 
                 sage: W.braid_orbit([2,1,1,2,1])
-                [[2, 2, 1, 2, 2], [2, 1, 1, 2, 1], [1, 2, 1, 1, 2], [2, 1, 2, 1, 2]]
+                [[2, 1, 1, 2, 1], [1, 2, 1, 1, 2], [2, 2, 1, 2, 2], [2, 1, 2, 1, 2]]
 
                 sage: W = ReflectionGroup(['A',3], index_set=["AA","BB",5])  # optional - gap3
                 sage: w = W.long_element()                                   # optional - gap3

comment:108 follow-up: Changed 3 months ago by tscrim

That is a big Sage-wide issue/question: should things be iterators/generators or lists. Some places we do both (e.g., in graphs), in others we only have the lists, and even still some places only do iterators.

The simple fix here to make everything consistent is just to sort the output (as was done in the test before) as this should be considered more as a set than an enumerated list IIRC.

comment:109 in reply to: ↑ 108 Changed 3 months ago by mkoeppe

Replying to tscrim:

The simple fix here to make everything consistent is just to sort the output (as was done in the test before) as this should be considered more as a set than an enumerated list IIRC.

Sure, this is what we should do for this ticket.

comment:110 Changed 3 months ago by mkoeppe

  • Dependencies changed from #28197, #29042, #29861 to #28197, #29042, #29861, #30162

comment:111 Changed 3 months ago by mkoeppe

  • Authors changed from Jeroen Demeyer to Jeroen Demeyer, John Palmieri, Matthias Koeppe

comment:112 follow-up: Changed 3 months ago by jhpalmieri

I don't understand the point of checking specific hash values. For example, in src/sage/combinat/similarity_class_type.py, the hashes depend on how Python determines hash(tuple([3,2,1])), for example. Since I think that is an implementation detail of Python, not Sage, doctesting for a specific value is the opposite of robust.

comment:113 in reply to: ↑ 112 Changed 3 months ago by mkoeppe

Replying to jhpalmieri:

I don't understand the point of checking specific hash values. For example, in src/sage/combinat/similarity_class_type.py, the hashes depend on how Python determines hash(tuple([3,2,1])), for example. Since I think that is an implementation detail of Python, not Sage, doctesting for a specific value is the opposite of robust.

My guess is that these tests are there to guard against thoughtless or accidental changes to hash functions.

I could imagine that we may want to change these tests to something like this:

    sage: hash(x) in (5050909583595644741, -925386691174542829, 234567876545678)
    True

comment:114 Changed 3 months ago by git

  • Commit changed from 5a5fa82344129e34f03b62ec2c3bb0a72de0d808 to 3d6ae39a62049a0a02e71f56c9cdd970407147a4

Branch pushed to git repo; I updated commit sha1. New commits:

3d6ae39trac 27754: fix some doctests for Python 3.8

comment:115 Changed 3 months ago by jhpalmieri

Here are "fixes" for some doctests. Some are probably not ideal. This is also far from complete: I didn't have the energy to do more, and I don't know how to handle some of the failures. But maybe this is a start.

comment:116 Changed 3 months ago by mkoeppe

  • Work issues set to merge #30162

comment:117 Changed 3 months ago by tscrim

For similarity_class_type.py, back when this was written IIRC, there was less decided about how to test a __hash__ method and less concern about how changes in Python would affect this. So I think the changed tests here are good.

Last edited 3 months ago by tscrim (previous) (diff)

comment:118 Changed 3 months ago by slelievre

  • Description modified (diff)

Python 3.8.5 was released as a security hotfix.

See #30184 for Python 3.9.x.

comment:119 follow-ups: Changed 3 months ago by jhpalmieri

I'm not going to be able to work on this for a few days. If someone else wants to work on doctests, please feel free. Do we have a Pillow-upgrade ticket? I wonder if that would help with some of the tests. And #30176 (matplotlib upgrade) might help, too.

Should this be delayed to Sage 9.3, ideally early in the release cycle?

comment:120 in reply to: ↑ 119 Changed 3 months ago by mkoeppe

Replying to jhpalmieri:

Do we have a Pillow-upgrade ticket?

Yes, it's on #30103

comment:121 Changed 3 months ago by heluani

  • Cc heluani added

comment:122 Changed 3 months ago by git

  • Commit changed from 3d6ae39a62049a0a02e71f56c9cdd970407147a4 to ccc461e59d246b87a99d8b3426ecd419bd8484ab

Branch pushed to git repo; I updated commit sha1. New commits:

8a43db6build/pkgs/python3: Update to 3.8.5
60a8922In doctests, accept both can't and cannot in error messages
71da37bFixup
2727aa8src/sage/combinat/words/word.py: More wildcarding for respelled error messages
c43343cEven more wildcarding for respelled error messages; remove some py2-only tests
1a7bc97Merge branch 'u/mkoeppe/make_doctests_accept_python_3_8_x_respellings_of_some_error_messages' of git://trac.sagemath.org/sage into t/27754/public/python3.8
ccc461eMerge branch 'public/python3.8' of git://trac.sagemath.org/sage into t/27754/public/python3.8

comment:123 Changed 3 months ago by mkoeppe

  • Summary changed from Upgrade: Python 3.8.4 to Upgrade: Python 3.8.5
  • Work issues merge #30162 deleted

comment:124 Changed 3 months ago by mkoeppe

  • Description modified (diff)

comment:125 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:126 in reply to: ↑ 119 Changed 3 months ago by mkoeppe

Replying to jhpalmieri:

Should this be delayed to Sage 9.3, ideally early in the release cycle?

Yes, I think so. Let's do #30210 (Upgrade Python to 3.7.8) for 9.2

comment:127 Changed 3 months ago by arojas

There are lots of failures caused by deprecation warnings on unpickling

DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats

that come from pynac. These are fixed with https://github.com/pynac/pynac/pull/354, but pynac development seems stalled unfortunately

comment:128 follow-up: Changed 3 months ago by mkoeppe

Could you please create a sage ticket for this pynac patch?

comment:129 Changed 3 months ago by git

  • Commit changed from ccc461e59d246b87a99d8b3426ecd419bd8484ab to b1fc3553e32333143adb763e74728b8d45d99f2e

Branch pushed to git repo; I updated commit sha1. New commits:

f243937Merge remote-tracking branch 'origin/develop' into t/27754/public/python3.8
73539f6Make tests in categories/finite_enumerated_sets.py pass with Python 3.8
0c893dbFix 'SyntaxWarning: "is" with a literal. Did you mean "=="?' with Python 3.8
b1fc355Make tests in combinat/words/word_infinite_datatypes.py compatible with Python 3.8

comment:130 in reply to: ↑ 128 Changed 3 months ago by arojas

  • Dependencies changed from #28197, #29042, #29861, #30162 to #28197, #29042, #29861, #30162, #30225

Replying to mkoeppe:

Could you please create a sage ticket for this pynac patch?

Done: #30225

comment:131 Changed 3 months ago by git

  • Commit changed from b1fc3553e32333143adb763e74728b8d45d99f2e to 6eb6a473d81c6bc3d04a3432f78ce3d8bfa192be

Branch pushed to git repo; I updated commit sha1. New commits:

6eb6a47More sorted()

comment:132 Changed 3 months ago by git

  • Commit changed from 6eb6a473d81c6bc3d04a3432f78ce3d8bfa192be to 583042aaac9c36e11e76a4c9accf95188ec9d2d5

Branch pushed to git repo; I updated commit sha1. New commits:

583042aUpdate compile flags for python 3.8

comment:133 Changed 3 months ago by git

  • Commit changed from 583042aaac9c36e11e76a4c9accf95188ec9d2d5 to d9ca5c5715da4b6c78adc66095eeaca7c4a26c96

Branch pushed to git repo; I updated commit sha1. New commits:

e573b93Port away from deprecated cgi.escape
d9ca5c5Fix deprecation warnings in graphs/graph_latex.py with Python 3.8

comment:134 Changed 3 months ago by arojas

  • Authors changed from Jeroen Demeyer, John Palmieri, Matthias Koeppe to Jeroen Demeyer, John Palmieri, Matthias Koeppe, Antonio Rojas

comment:135 Changed 3 months ago by git

  • Commit changed from d9ca5c5715da4b6c78adc66095eeaca7c4a26c96 to 86d5e94dc2ce2f0b674b275e039cecaaa7d0e130

Branch pushed to git repo; I updated commit sha1. New commits:

86d5e94Rewrite some hash tests to be independent of the actual hashes

comment:136 Changed 3 months ago by git

  • Commit changed from 86d5e94dc2ce2f0b674b275e039cecaaa7d0e130 to be98eb0a2721506f55a1a9dd30a647d006018e01

Branch pushed to git repo; I updated commit sha1. New commits:

be98eb0Rewrite direct tests for hashes

comment:137 Changed 3 months ago by git

  • Commit changed from be98eb0a2721506f55a1a9dd30a647d006018e01 to 74f9352dccdd1d7e4f2c21da22b6625b88362884

Branch pushed to git repo; I updated commit sha1. New commits:

cdd8cb6Homogenize tuple notation from previous commit
74f9352Add posonlyargcount argument required by Python 3.8

comment:138 Changed 3 months ago by git

  • Commit changed from 74f9352dccdd1d7e4f2c21da22b6625b88362884 to a736d791ddeeea6dbc2e53e2c34a580dda32127b

Branch pushed to git repo; I updated commit sha1. New commits:

ea55182Accept more detailed syntax error with Python 3.8
a736d79Fix missing import

comment:139 Changed 3 months ago by git

  • Commit changed from a736d791ddeeea6dbc2e53e2c34a580dda32127b to f384065cbf40774888040b1180ed0320add546ad

Branch pushed to git repo; I updated commit sha1. New commits:

f384065Make computing graph faces deterministic

comment:140 Changed 3 months ago by git

  • Commit changed from f384065cbf40774888040b1180ed0320add546ad to 694c6b153eb1b2e69aa5345f5108ceeb6acfb1d9

Branch pushed to git repo; I updated commit sha1. New commits:

de57122Update tests in graphs/schnyder.py after previous commit
694c6b1Cast sleep arguments to float

comment:141 Changed 3 months ago by git

  • Commit changed from 694c6b153eb1b2e69aa5345f5108ceeb6acfb1d9 to 14c57e57275bdedce87b159d4cfe9658e94ec555

Branch pushed to git repo; I updated commit sha1. New commits:

3597904Accept ValueError in addition to KeyError
14c57e5More sorted()

comment:142 Changed 3 months ago by git

  • Commit changed from 14c57e57275bdedce87b159d4cfe9658e94ec555 to cd346fa5bf72e14338a944626aa6fc3e8fe2833c

Branch pushed to git repo; I updated commit sha1. New commits:

446afe7Remove some randomness
cd346faMake computing fan rays deterministic

comment:143 Changed 3 months ago by git

  • Commit changed from cd346fa5bf72e14338a944626aa6fc3e8fe2833c to 093e9d61181d5ae221bc7d20e4b408b7efb251e8

Branch pushed to git repo; I updated commit sha1. New commits:

feb582aMake sparse_2term_quotient deterministic
093e9d6More sorting

comment:144 Changed 3 months ago by git

  • Commit changed from 093e9d61181d5ae221bc7d20e4b408b7efb251e8 to 73d6c7f47c6a203da2cca58f56e68df6fc3e6852

Branch pushed to git repo; I updated commit sha1. New commits:

9ee9cb8Fix some tests
72b40caRevert unintended change
73d6c7fUpdate test

comment:145 Changed 3 months ago by arojas

Just a handful of issues left, mostly pickling related

**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/matroids/linear_matroid.pyx", line 3674, in sage.matroids.linear_matroid.BinaryMatroid._projection_partition
Failed example:
    N._projection_partition()
Expected:
    2 x 12 BinaryMatrix
    [110011001100]
    [001100110011]
Got:
    2 x 12 BinaryMatrix
    [001100110011]
    [110011001100]
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/structure/sequence.py", line 824, in sage.structure.sequence.Sequence_generic.__copy__
Failed example:
    t.is_immutable == s.is_immutable
Expected:
    True
Got:
    False
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/structure/sequence.py", line 826, in sage.structure.sequence.Sequence_generic.__copy__
Failed example:
    t.is_mutable == s.is_mutable
Expected:
    True
Got:
    False
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/combinat/posets/hasse_diagram.py", line 2185, in sage.combinat.posets.hasse_diagram.HasseDiagram.antichains
Failed example:
    TestSuite(A).run()
Expected nothing
Got:
    Failure in _test_pickling:
    Traceback (most recent call last):
      File "/usr/lib/python3.8/site-packages/sage/misc/sage_unittest.py", line 296, in run
        test_method(tester=tester)
      File "sage/structure/sage_object.pyx", line 639, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:4953)
        tester.assertEqual(loads(dumps(self)), self)
      File "/usr/lib/python3.8/unittest/case.py", line 912, in assertEqual
        assertion_func(first, second, msg=msg)
      File "/usr/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
        raise self.failureException(msg)
    AssertionError: An enumerated set with a forest structure != An enumerated set with a forest structure
    ------------------------------------------------------------
    The following tests failed: _test_pickling
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/combinat/posets/hasse_diagram.py", line 2190, in sage.combinat.posets.hasse_diagram.HasseDiagram.antichains
Failed example:
    TestSuite(A).run()
Expected nothing
Got:
    Failure in _test_pickling:
    Traceback (most recent call last):
      File "/usr/lib/python3.8/site-packages/sage/misc/sage_unittest.py", line 296, in run
        test_method(tester=tester)
      File "sage/structure/sage_object.pyx", line 639, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:4953)
        tester.assertEqual(loads(dumps(self)), self)
      File "/usr/lib/python3.8/unittest/case.py", line 912, in assertEqual
        assertion_func(first, second, msg=msg)
      File "/usr/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
        raise self.failureException(msg)
    AssertionError: An enumerated set with a forest structure != An enumerated set with a forest structure
    ------------------------------------------------------------
    The following tests failed: _test_pickling
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/combinat/cluster_algebra_quiver/quiver_mutation_type.py", line 2171, in sage.combinat.cluster_algebra_quiver.quiver_mutation_type._construct_exceptional_mutation_classes
Failed example:
    for mut_class in sorted(rank_3_exceptional.keys(), key=str): # long time
      print("{} {}".format(mut_class, rank_3_exceptional[mut_class]))
Expected:
    ('G', 2, -1) [('BH?', (((1, 2), (1, -3)),)),
    ('BGO', (((2, 1), (3, -1)),)), ('BW?', (((0, 1), (3, -1)),)),
    ('BP?', (((0, 1), (1, -3)),)),
    ('BP_', (((0, 1), (1, -3)), ((2, 0), (3, -1)))),
    ('BP_', (((0, 1), (3, -1)), ((1, 2), (1, -3)), ((2, 0), (2, -2))))]
    ('G', 2, 1) [('BH?', (((1, 2), (3, -1)),)),
    ('BGO', (((2, 1), (1, -3)),)), ('BW?', (((0, 1), (1, -3)),)),
    ('BP?', (((0, 1), (3, -1)),)),
    ('BKO', (((1, 0), (3, -1)), ((2, 1), (1, -3)))),
    ('BP_', (((0, 1), (2, -2)), ((1, 2), (1, -3)), ((2, 0), (3, -1))))]
Got:
    ('G', 2, -1) [('BH?', (((1, 2), (1, -3)),)), ('BGO', (((2, 1), (3, -1)),)), ('BW?', (((0, 1), (3, -1)),)), ('BP?', (((0, 1), (1, -3)),)), ('BP_', (((0, 1), (1, -3)), ((2, 0), (3, -1)))), ('BP_', (((0, 1), (1, -3)), ((1, 2), (2, -2)), ((2, 0), (3, -1))))]
    ('G', 2, 1) [('BH?', (((1, 2), (3, -1)),)), ('BGO', (((2, 1), (1, -3)),)), ('BW?', (((0, 1), (1, -3)),)), ('BP?', (((0, 1), (3, -1)),)), ('BKO', (((1, 0), (3, -1)), ((2, 1), (1, -3)))), ('BP_', (((0, 1), (3, -1)), ((1, 2), (2, -2)), ((2, 0), (1, -3)))), ('BP_', (((0, 1), (2, -2)), ((1, 2), (1, -3)), ((2, 0), (3, -1))))]
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/combinat/cluster_algebra_quiver/mutation_type.py", line 1348, in sage.combinat.cluster_algebra_quiver.mutation_type._mutation_type_test
Failed example:
    _mutation_type_test(3) # long time
Expected:
    True ('A', (2, 1), 1)
    True ('A', 3)
    True ('B', 3)
    True ('BB', 2, 1)
    True ('BC', 2, 1)
    True ('C', 3)
    True ('CC', 2, 1)
    True ('G', 2, -1)
    True ('G', 2, 1)
Got:
    True ('A', (2, 1), 1)
    True ('A', 3)
    True ('B', 3)
    True ('BB', 2, 1)
    True ('BC', 2, 1)
    True ('C', 3)
    True ('CC', 2, 1)
    False ('G', 2, -1)
    False ('G', 2, 1)
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/schemes/riemann_surfaces/riemann_surface.py", line 355, in sage.schemes.riemann_surfaces.riemann_surface.RiemannSurface
Failed example:
    S.riemann_matrix() #abs tol 0.00000001
Expected:
    [0.500000000000000000000000... + 0.866025403784438646763723...*I]
Got:
    [-0.50000000000000000000000000000 + 0.86602540378443864676372317075*I]
Tolerance exceeded in 1 of 2:
    0.500000000000000000000000 vs -0.50000000000000000000000000000, tolerance 1e0 > 1e-8
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/homology/simplicial_complex_morphism.py", line 82, in sage.homology.simplicial_complex_morphism
Failed example:
    z                                     # this is the mapping path space
Expected:
    Simplicial complex morphism:
      From: Simplicial complex with 6 vertices and 4 facets
      To:   Minimal triangulation of the 2-sphere
      Defn: ['L0R(0, 0)', 'L0R(0, 1)', 'L1R(1, 0)', 'L1R(1, 1)', 'L2R(2, 0)', 'L2R(2, 1)'] --> [0, 0, 1, 1, 2, 2]
Got:
    Simplicial complex morphism:
      From: Simplicial complex with 6 vertices and 5 facets
      To:   Minimal triangulation of the 2-sphere
      Defn: ['L0R(0, 0)', 'L0R(0, 1)', 'L1R(1, 0)', 'L1R(1, 1)', 'L2R(2, 0)', 'L2R(2, 1)'] --> [0, 0, 1, 1, 2, 2]
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/categories/homset.py", line 598, in sage.categories.homset.Homset.__init__
Failed example:
    TestSuite(H).run()
Expected nothing
Got:
      Failure in _test_pickling:
      Traceback (most recent call last):
        File "/usr/lib/python3.8/site-packages/sage/misc/sage_unittest.py", line 296, in run
          test_method(tester=tester)
        File "sage/structure/sage_object.pyx", line 639, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:4953)
          tester.assertEqual(loads(dumps(self)), self)
        File "/usr/lib/python3.8/unittest/case.py", line 912, in assertEqual
          assertion_func(first, second, msg=msg)
        File "/usr/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
          raise self.failureException(msg)
      AssertionError: Generic morphism:
        From: X
        To:   Y != Generic morphism:
        From: X
        To:   Y
      ------------------------------------------------------------
      The following tests failed: _test_pickling
    Failure in _test_elements
    The following tests failed: _test_elements
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/categories/homset.py", line 607, in sage.categories.homset.Homset.__init__
Failed example:
    TestSuite(H).run()
Expected nothing
Got:
      Failure in _test_pickling:
      Traceback (most recent call last):
        File "/usr/lib/python3.8/site-packages/sage/misc/sage_unittest.py", line 296, in run
          test_method(tester=tester)
        File "sage/structure/sage_object.pyx", line 639, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:4953)
          tester.assertEqual(loads(dumps(self)), self)
        File "/usr/lib/python3.8/unittest/case.py", line 912, in assertEqual
          assertion_func(first, second, msg=msg)
        File "/usr/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
          raise self.failureException(msg)
      AssertionError: Generic morphism:
        From: X
        To:   Y != Generic morphism:
        From: X
        To:   Y
      ------------------------------------------------------------
      The following tests failed: _test_pickling
    Failure in _test_elements
    The following tests failed: _test_elements
**********************************************************************
File "/usr/lib/python3.8/site-packages/sage/categories/homset.py", line 1253, in sage.categories.homset.HomsetWithBase.__init__
Failed example:
    TestSuite(H).run()
Expected nothing
Got:
      Failure in _test_pickling:
      Traceback (most recent call last):
        File "/usr/lib/python3.8/site-packages/sage/misc/sage_unittest.py", line 296, in run
          test_method(tester=tester)
        File "sage/structure/sage_object.pyx", line 639, in sage.structure.sage_object.SageObject._test_pickling (build/cythonized/sage/structure/sage_object.c:4953)
          tester.assertEqual(loads(dumps(self)), self)
        File "/usr/lib/python3.8/unittest/case.py", line 912, in assertEqual
          assertion_func(first, second, msg=msg)
        File "/usr/lib/python3.8/unittest/case.py", line 905, in _baseAssertEqual
          raise self.failureException(msg)
      AssertionError: Generic morphism:
        From: X
        To:   Y != Generic morphism:
        From: X
        To:   Y
      ------------------------------------------------------------
      The following tests failed: _test_pickling
    Failure in _test_elements
    The following tests failed: _test_elements

comment:146 Changed 3 months ago by git

  • Commit changed from 73d6c7f47c6a203da2cca58f56e68df6fc3e6852 to ce0f0e0404b23485c7c8b2e38e4c8a36eb1354e1

Branch pushed to git repo; I updated commit sha1. New commits:

687d070Make vertices tuple of a simplicial complex deterministic
46114aaMake test in matroids/linear_matroid.pyx pass regardless of the matrix row order
723b48fMake _graph_without_edge_labels deterministic
ce0f0e0Make riemann_matrix test more robust

comment:147 Changed 3 months ago by arojas

The remaining issues are caused by this commit https://github.com/python/cpython/commit/ac20e0f98d6727ba97a9575bfa2a11b2f6247c35

In particular, if B=loads(dumps(A)), then A.method and B.method have different hashes, so A.method==B.method is always false.

comment:148 follow-up: Changed 3 months ago by jhpalmieri

Note that the issue of sorting vertices in simplicial complexes was discussed at length in #26966. I may propose a different solution to this problem, but give me a few days.

comment:149 Changed 3 months ago by git

  • Commit changed from ce0f0e0404b23485c7c8b2e38e4c8a36eb1354e1 to 5a98f9094bf29c5ddb7098ea21ca7076ab6d1a7d

Branch pushed to git repo; I updated commit sha1. New commits:

a4d7660Call methods before comparing
f8768cfCompare the string representations of predicated instead of the predicates themselves
8369b60Fix tests in categories/homset.py
5a98f90Remove duplicate tests

comment:150 in reply to: ↑ 148 Changed 3 months ago by arojas

Replying to jhpalmieri:

Note that the issue of sorting vertices in simplicial complexes was discussed at length in #26966. I may propose a different solution to this problem, but give me a few days.

Oh, I see now that you proposed the same fix there but reverted it.

comment:151 Changed 3 months ago by arojas

With the current branch all tests are passing for me. There may be room for some improvements (besides the simplicial complex vertex ordering issue), but I hope there's still a chance that this can make it in 9.2. It would be quite unfortunate to not support python 3.8 one year after its release.

comment:152 Changed 3 months ago by jhpalmieri

I wrote a long post about simplicial complexes with various solutions, but now I realize there is a simpler solution.

There are a few problems here: first, if you want consistency when dealing with products of simplicial complexes, some sorting is needed: if you triangulate a square, for example, you have two choices for how to add the diagonal, and different sortings will result in different choices. Second, Python 3.7 and 3.8 are handling some data differently from each other, and that's where the doctest failure in simplicial_complex_morphism.py is coming from: different sortings yielding different triangulations of a product. Third, I think that this really means that the doctest in simplicial_complex_morphishm.py is a bad one, because it is essentially relying on how intrinsically unsorted data is converted to an ordered object (a tuple). I tried putting in various ways of sorting, and I got at least three different results, all of which are mathematically valid, and I don't see any reason to force one sorting option over another.

As a result, we should make this change: get rid of the new sorting in simplicial_complex.py and make the doctest more robust:

  • src/sage/homology/simplicial_complex.py

    diff --git a/src/sage/homology/simplicial_complex.py b/src/sage/homology/simplicial_complex.py
    index 8ee9a4f84c..1c9dd8da8e 100644
    a b class SimplicialComplex(Parent, GenericCellComplex): 
    10451045        else:
    10461046            vertex_set = range(n + 1)
    10471047
    1048         try:               # use natural order if vertices are integers
    1049             vertices = tuple(sorted(vertex_set))
    1050         except TypeError:  # incomparable vertices
    1051             vertices = tuple(sorted(vertex_set, key=str))
     1048        vertices = tuple(vertex_set)
    10521049
    10531050        gen_dict = {}
    10541051        for v in vertices:
  • src/sage/homology/simplicial_complex_morphism.py

    diff --git a/src/sage/homology/simplicial_complex_morphism.py b/src/sage/homology/simplicial_complex_morphism.py
    index d954dd90b4..8db933b9fe 100644
    a b EXAMPLES:: 
    8181    sage: z = y.fiber_product(x)
    8282    sage: z                                     # this is the mapping path space
    8383    Simplicial complex morphism:
    84       From: Simplicial complex with 6 vertices and 6 facets
     84      From: Simplicial complex with 6 vertices and ... facets
    8585      To:   Minimal triangulation of the 2-sphere
    8686      Defn: ['L0R(0, 0)', 'L0R(0, 1)', 'L1R(1, 0)', 'L1R(1, 1)', 'L2R(2, 0)', 'L2R(2, 1)'] --> [0, 0, 1, 1, 2, 2]
    8787"""

By the way, see ticket:26966#comment:25 for the an example of the evils of sorting using key=str.

comment:153 Changed 3 months ago by git

  • Commit changed from 5a98f9094bf29c5ddb7098ea21ca7076ab6d1a7d to 196a30ad097999a707b71140abc6afd5265c652f

Branch pushed to git repo; I updated commit sha1. New commits:

196a30atrac 27754: make doctest in simplicial_complex_morphism.py more robust

comment:154 Changed 3 months ago by git

  • Commit changed from 196a30ad097999a707b71140abc6afd5265c652f to 35cf0e197e4142b4849d09398a31063b5537eec9

Branch pushed to git repo; I updated commit sha1. New commits:

35cf0e1Allow system python 3.8

comment:155 Changed 3 months ago by git

  • Commit changed from 35cf0e197e4142b4849d09398a31063b5537eec9 to f939f28b40da12dfc5b31f531dac380f5ef20343

Branch pushed to git repo; I updated commit sha1. New commits:

10d46f4Merge branch 'develop' of git://git.sagemath.org/sage into t/27754/public/python3.8
f939f28Fix one more SyntaxWarning: 'is' with a literal. Did you mean '=='?, not caught by doctests

comment:156 Changed 3 months ago by arojas

  • Status changed from new to needs_review

Setting to needs review for wider testing. Needs testing especially with system python<3.8, also with optional packages

comment:157 Changed 3 months ago by git

  • Commit changed from f939f28b40da12dfc5b31f531dac380f5ef20343 to c6002e8181f8b4bf2b9a4f2262a60aa354c16e4f

Branch pushed to git repo; I updated commit sha1. New commits:

c4769bfAdd pynac patch to fix deprecation warnings on unpickling with Python 3.8
ecc6f45Add patch
c6002e8Merge branch 'u/arojas/fix_deprecation_warnings_when_unpickling_pynac_objects_with_python_3_8' of git://trac.sagemath.org/sage into t/27754/public/python3.8

comment:159 Changed 3 months ago by mkoeppe

  • Cc gh-sheerluck added

comment:160 Changed 3 months ago by git

  • Commit changed from c6002e8181f8b4bf2b9a4f2262a60aa354c16e4f to 57c2d1c89515fe0716d04159bcdbe3fc398e1180

Branch pushed to git repo; I updated commit sha1. New commits:

57c2d1ctox.ini, .github/workflows/tox.yml: Add gentoo-python3.7

comment:161 in reply to: ↑ 158 Changed 3 months ago by arojas

Replying to mkoeppe:

Tests run at https://github.com/mkoeppe/sage/actions/runs/193151775

What's the outcome of this? I see many failed jobs but I can't see any logs.

comment:162 Changed 3 months ago by mkoeppe

Looks like GH Action was having trouble. I have started another run at https://github.com/mkoeppe/sage/actions/runs/199465086

comment:163 Changed 3 months ago by mkoeppe

Clean on ubuntu-trusty-standard (with python3 from SPKG, https://github.com/mkoeppe/sage/runs/962936559), except for

sage -t --random-seed=0 src/sage/doctest/test.py  # Timed out

Clean on centos-7-standard (with python3 from SPKG, https://github.com/mkoeppe/sage/runs/962936782), except for

sage -t --random-seed=0 src/sage/interfaces/sagespawn.pyx
File "src/sage/interfaces/sagespawn.pyx", line 96, in sage.interfaces.sagespawn.SageSpawn.__repr__
Failed example:
    while s.isalive():  # Wait until the process finishes
        sleep(0.1)
Expected nothing
Got:
    doctest:warning
    DeprecationWarning: an integer is required (got type sage.rings.real_mpfr.RealLiteral).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.

Clean on gentoo-standard (using /usr/bin/python3.8, https://github.com/mkoeppe/sage/runs/962936791) except for:

sage -t --random-seed=0 src/sage/schemes/elliptic_curves/ell_modular_symbols.py  # Timed out

Clean on gentoo-standard-python3.7 (using /usr/bin/python3.7) except for:

sage -t --random-seed=0 src/sage/categories/finite_enumerated_sets.py
File "src/sage/categories/finite_enumerated_sets.py", line 290, in sage.categories.finite_enumerated_sets.FiniteEnumeratedSets.ParentMethods._list_from_iterator
Failed example:
    list(C)
Expected:
    hello!
    hello!
    ...
    [1, 2, 3]
Got:
    hello!
    hello!
    [1, 2, 3]

Lots of doctest failures because of LC_ALL: cannot change locale messages (#30053) on linuxmint-17-minimal (https://github.com/mkoeppe/sage/runs/962936711) and -standard

Unfortunately most builds failed again because of unrelated technical problems on GH Actions

comment:164 Changed 3 months ago by mkoeppe

  • Reviewers set to https://github.com/mkoeppe/sage/actions/runs/199465086, https://github.com/mkoeppe/sage/runs/959545895

comment:165 Changed 3 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/actions/runs/199465086, https://github.com/mkoeppe/sage/runs/959545895 to https://github.com/mkoeppe/sage/actions/runs/199465086, https://github.com/mkoeppe/sage/actions/runs/199465096

comment:166 Changed 3 months ago by git

  • Commit changed from 57c2d1c89515fe0716d04159bcdbe3fc398e1180 to 389f1b2d7f5fe78ed232bbfb50d95ca5bee47d1e

Branch pushed to git repo; I updated commit sha1. New commits:

a5f870bConvert sleep() argument to float
625dc9dMake test pass with python 3.7
389f1b2Merge branch 'public/python3.8' of git://trac.sagemath.org/sage into t/27754/public/python3.8

comment:167 Changed 3 months ago by git

  • Commit changed from 389f1b2d7f5fe78ed232bbfb50d95ca5bee47d1e to fa2ad16159d452089a235796291a13f49d8d0477

Branch pushed to git repo; I updated commit sha1. New commits:

fa2ad16Merge tag '9.2.beta8' into t/27754/public/python3.8

comment:168 Changed 3 months ago by mkoeppe

  • Reviewers changed from https://github.com/mkoeppe/sage/actions/runs/199465086, https://github.com/mkoeppe/sage/actions/runs/199465096 to github.com/mkoeppe/sage/actions/runs/206442118

comment:169 Changed 3 months ago by mkoeppe

debian-jessie-minimal, debian-stretch-standard, centos-7-standard (with python3 from SPKG) clean

ubuntu-bionic-standard, debian-buster-standard, linuxmint-19.3-standard, fedora-30-standard, fedora-31-standard, gentoo-python3.7-standard (with /usr/bin/python3.7) clean

gentoo-standard (with /usr/bin/python3.8) clean

ubuntu-eoan-standard (with /usr/bin/python3.7, https://github.com/mkoeppe/sage/runs/978617270) clean except for

sage -t --random-seed=0 src/sage/libs/eclib/interface.py  # 2 doctests failed
sage -t --random-seed=0 src/sage/schemes/cyclic_covers/charpoly_frobenius.py  # 1 doctest failed

(seem unrelated to this ticket (mwrank; #28951?))

debian-bullseye-standard (with /usr/bin/python3.7) clean except for doctest errors because of newer flint or arb from the system

linuxmint-17-standard .... lots of doctest failures because of "setlocale" messages

fedora-27-standard (with /usr/bin/python3.6):

sage -t --random-seed=0 src/sage/lfunctions/sympow.py  # 3 doctests failed
sage -t --random-seed=0 src/sage/schemes/elliptic_curves/ell_rational_field.py  # 2 doctests failed

(seems unrelated)

fedora-29-standard (with /usr/bin/python3.7): likewise

fedora-28-standard (with /usr/bin/python3.6) additionally:

sage -t --random-seed=0 src/sage/rings/polynomial/groebner_fan.py  # 3 doctests failed

(seems unrelated)

conda-forge-standard (with /opt/conda/bin/python3.7):

sage -t --random-seed=0 src/doc/en/constructions/algebraic_geometry.rst  # 1 doctest failed
sage -t --random-seed=0 src/doc/en/developer/coding_in_other.rst  # 1 doctest failed
sage -t --random-seed=0 src/sage/misc/cython.py  # 3 doctests failed
sage -t --random-seed=0 src/sage/modules/fg_pid/fgp_module.py  # 3 doctests failed
sage -t --random-seed=0 src/sage/modules/free_module_morphism.py  # 3 doctests failed
sage -t --random-seed=0 src/sage/rings/complex_arb.pyx  # 6 doctests failed
sage -t --random-seed=0 src/sage/parallel/map_reduce.py  # Timed out
sage -t --random-seed=0 src/sage/rings/real_arb.pyx  # 2 doctests failed
sage -t --random-seed=0 src/sage/schemes/hyperelliptic_curves/monsky_washnitzer.py  # Killed due to abort

conda-forge-minimal (https://github.com/mkoeppe/sage/runs/978620286):

  [python3-3.8.5]   ImportError: The required _crypt module was not built as part of CPython
  [python3-3.8.5]   crypt module failed to import
  [python3-3.8.5]   ModuleNotFoundError: No module named 'zlib'
  [python3-3.8.5]   zlib module failed to import
  [python3-3.8.5]   ModuleNotFoundError: No module named '_sqlite3'
  [python3-3.8.5]   sqlite3 module failed to import
  [python3-3.8.5]   Error: One or more modules failed to import.

(but we had similar errors also before the update, on this platform)

ubuntu-focal-standard (https://github.com/mkoeppe/sage/runs/978617297)

hecking whether SageMath should install SPKG python3...
checking whether any of sqlite libpng bzip2 xz libffi is installed as or will be installed as SPKG... no
checking for python3 >= 3.6, < 3.9 with modules sqlite3, ctypes, math, hashlib, crypt, readline, socket, zlib, distutils.core... 
checking ... whether /usr/bin/python3.8 is good... no, the version is in the supported range, and the modules can be imported, but distutils cannot build a C extension
checking ... whether /usr/bin/python3 is good... no, the version is in the supported range, and the modules can be imported, but distutils cannot build a C extension
checking ... whether /bin/python3.8 is good... no, the version is in the supported range, and the modules can be imported, but distutils cannot build a C extension
checking ... whether /bin/python3 is good... no, the version is in the supported range, and the modules can be imported, but distutils cannot build a C extension

configure: no suitable system package found for SPKG python3

likewise on debian-sid-standard (https://github.com/mkoeppe/sage/runs/978620025)

Last edited 3 months ago by mkoeppe (previous) (diff)

comment:170 Changed 3 months ago by mkoeppe

homebrew-macos-minimal (using python3 SPKG): Clean except for various timeouts

sage -t --random-seed=0 src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # Timed out
sage -t --random-seed=0 src/sage/interfaces/gap.py  # 1 doctest failed
sage -t --random-seed=0 src/sage/manifolds/differentiable/metric.py  # Timed out
sage -t --random-seed=0 src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t --random-seed=0 src/sage/plot/graphics.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/base.pyx  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/implicit_plot3d.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/parametric_plot3d.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/plot3d.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/shapes.pyx  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/shapes2.py  # Timed out
sage -t --random-seed=0 src/sage/rings/function_field/function_field.py  # Timed out
sage -t --random-seed=0 src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out
sage -t --random-seed=0 src/sage/schemes/elliptic_curves/gp_simon.py  

homebrew-macos-python3_xcode-standard (using /usr/bin/python3) -- build times out during docbuild --- perhaps the same problem as reported on sage-release

conda-forge-macos-minimal (https://github.com/mkoeppe/sage/runs/978617180):

  [python3-3.8.5]   checking build system type... x86_64-apple-darwin19.6.0
  [python3-3.8.5]   checking host system type... x86_64-apple-darwin13.4.0
  [python3-3.8.5]   checking for python3.8... no
  [python3-3.8.5]   checking for python3... python3
  [python3-3.8.5]   configure: error: Cross compiling required --host=HOST-TUPLE and --build=ARCH
  [python3-3.8.5]   configure: WARNING: unrecognized options: --disable-toolbox-glue
  [python3-3.8.5]   checking build system type... x86_64-apple-darwin19.6.0
  [python3-3.8.5]   checking host system type... x86_64-apple-darwin13.4.0
  [python3-3.8.5]   checking for python3.8... no
  [python3-3.8.5]   checking for python3... python3
  [python3-3.8.5]   configure: error: Cross compiling required --host=HOST-TUPLE and --build=ARCH

???? But this platform also failed before the update

Last edited 3 months ago by mkoeppe (previous) (diff)

comment:171 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.2

Looks like we have a good chance of getting this ready for 9.2!

comment:172 Changed 3 months ago by mkoeppe

(deleted)

Last edited 3 months ago by mkoeppe (previous) (diff)

comment:173 Changed 3 months ago by git

  • Commit changed from fa2ad16159d452089a235796291a13f49d8d0477 to 89dc8c92ebd9f10c7d41e6b72e6d271e05fc1119

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

bccf5c1Merge branch 't/30127/cygwin_minimal__pip_20_1_1_fails_to_install' into t/27754/public/python3.8
bc65004tox.ini, tox.yml: Add ubuntu-groovy, fedora-33
8eb9caeMerge branch 't/30339/tox_ini__update_for_latest_ubuntu__fedora_versions' into t/27754/public/python3.8
8d5f2aetox.ini, build/pkgs/python3/distros/: Switch to python 3.8
cf25b83Move code for tagging/pushing docker images into tox.ini
51c6cdefixup
231b812.github-workflows/tox-*.yml: Update similarly
385571eMerge branch 't/30144/tox_ini___tox_yml__move_code_for_tagging_pushing_docker_images_into_tox_ini' into t/27754/public/python3.8
701b4cebuild/make/Makefile.in: Add + before recursive make invocations
89dc8c9Merge branch 't/30345/build_make_makefile_in__filter_out___j__from_sub_make_invocations_to_avoid_excessive_parallel_load' into t/27754/public/python3.8

comment:174 Changed 3 months ago by mkoeppe

  • Dependencies changed from #28197, #29042, #29861, #30162, #30225 to #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345

comment:175 Changed 3 months ago by mkoeppe

  • Reviewers changed from github.com/mkoeppe/sage/actions/runs/206442118 to github.com/mkoeppe/sage/actions/runs/206442118, github.com/mkoeppe/sage/actions/runs/207801205, github.com/mkoeppe/sage/actions/runs/207801208, github.com/mkoeppe/sage/actions/runs/207801210

comment:176 Changed 3 months ago by mkoeppe

  • Reviewers changed from github.com/mkoeppe/sage/actions/runs/206442118, github.com/mkoeppe/sage/actions/runs/207801205, github.com/mkoeppe/sage/actions/runs/207801208, github.com/mkoeppe/sage/actions/runs/207801210 to github.com/mkoeppe/sage/actions/runs/207801205, github.com/mkoeppe/sage/actions/runs/207801208, github.com/mkoeppe/sage/actions/runs/207801210

comment:177 Changed 3 months ago by mkoeppe

  • Priority changed from major to critical

comment:178 Changed 3 months ago by mkoeppe

ubuntu-focal-standard now accepts system python 3.8, clean doctests.

But:

debian-bullseye-standard (https://github.com/mkoeppe/sage/runs/982720084), now using /usr/bin/python3.8 (3.8.5), fails to build matplotlib:

  x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-rpath-link,/sage/local/lib -L/sage/local/lib -Wl,-rpath,/sage/local/lib -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.8/src/qhull_wrap.o build/temp.linux-x86_64-3.8/extern/libqhull/geom.o build/temp.linux-x86_64-3.8/extern/libqhull/geom2.o build/temp.linux-x86_64-3.8/extern/libqhull/global.o build/temp.linux-x86_64-3.8/extern/libqhull/io.o build/temp.linux-x86_64-3.8/extern/libqhull/libqhull.o build/temp.linux-x86_64-3.8/extern/libqhull/mem.o build/temp.linux-x86_64-3.8/extern/libqhull/merge.o build/temp.linux-x86_64-3.8/extern/libqhull/poly.o build/temp.linux-x86_64-3.8/extern/libqhull/poly2.o build/temp.linux-x86_64-3.8/extern/libqhull/qset.o build/temp.linux-x86_64-3.8/extern/libqhull/random.o build/temp.linux-x86_64-3.8/extern/libqhull/rboxlib.o build/temp.linux-x86_64-3.8/extern/libqhull/stat.o build/temp.linux-x86_64-3.8/extern/libqhull/user.o build/temp.linux-x86_64-3.8/extern/libqhull/usermem.o build/temp.linux-x86_64-3.8/extern/libqhull/userprintf.o build/temp.linux-x86_64-3.8/extern/libqhull/userprintf_rbox.o -lm -o build/lib.linux-x86_64-3.8/matplotlib/_qhull.cpython-38-x86_64-linux-gnu.so
  lto1: fatal error: bytecode stream in file ‘build/temp.linux-x86_64-3.8/src/qhull_wrap.o’ generated with GCC compiler older than 10.0
  compilation terminated.
  lto-wrapper: fatal error: x86_64-linux-gnu-gcc returned 1 exit status
  compilation terminated.
  /usr/bin/ld: error: lto-wrapper failed
  collect2: error: ld returned 1 exit status

Likewise on debian-sid-standard and ubuntu-groovy-standard

Last edited 3 months ago by mkoeppe (previous) (diff)

comment:179 follow-up: Changed 3 months ago by mkoeppe

Getting rid of -flto would work around this problem. Not sure where it is coming from

comment:180 Changed 2 months ago by mkoeppe

(As expected, upgrading matplotlib to latest (3.3.1, #30358) does not solve this problem.)

comment:181 follow-up: Changed 2 months ago by fbissey

I haven't experienced anything like that in Gentoo. But looking at build log for matplotlib 3.3.0 suggests reviving an idea: using an external qhull install for matplotlib.

comment:182 in reply to: ↑ 179 Changed 2 months ago by mkoeppe

Replying to mkoeppe:

Getting rid of -flto would work around this problem. Not sure where it is coming from

The flag is coming directly from matplotlib's setup.py, I'll just disable it

comment:183 Changed 2 months ago by mkoeppe

  • Dependencies changed from #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345 to #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365

comment:184 Changed 2 months ago by git

  • Commit changed from 89dc8c92ebd9f10c7d41e6b72e6d271e05fc1119 to 4daa290fef4356faaf56eefa4e58b948e2f8db84

Branch pushed to git repo; I updated commit sha1. New commits:

2d16e5dbuild/pkgs/matplotlib/make-setup-config.py: Disable LTO
4daa290Merge branch 't/30365/disable_lto_in_matplotlib_build' into t/27754/public/python3.8

comment:185 in reply to: ↑ 181 Changed 2 months ago by mkoeppe

Replying to fbissey:

I haven't experienced anything like that in Gentoo. But looking at build log for matplotlib 3.3.0 suggests reviving an idea: using an external qhull install for matplotlib.

I would support this in general, but not for 9.2; and as discussed in #30176, we would need to make cmake standard.

comment:186 Changed 2 months ago by mkoeppe

  • Reviewers changed from github.com/mkoeppe/sage/actions/runs/207801205, github.com/mkoeppe/sage/actions/runs/207801208, github.com/mkoeppe/sage/actions/runs/207801210 to Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/207801205, github.com/mkoeppe/sage/actions/runs/207801208, github.com/mkoeppe/sage/actions/runs/207801210, ...

I think this ticket is now ready for review.

comment:187 Changed 2 months ago by mkoeppe

  • Description modified (diff)

comment:188 Changed 2 months ago by fbissey

  • Reviewers changed from Matthias Koeppe, github.com/mkoeppe/sage/actions/runs/207801205, github.com/mkoeppe/sage/actions/runs/207801208, github.com/mkoeppe/sage/actions/runs/207801210, ... to Matthias Koeppe, François Bissey
  • Status changed from needs_review to positive_review

Now is a good time to tests things on a bigger scale.

comment:189 Changed 2 months ago by mkoeppe

Thanks

comment:190 Changed 2 months ago by git

  • Commit changed from 4daa290fef4356faaf56eefa4e58b948e2f8db84 to 46c559c0b052e37d87c050fee879150318ba0463
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

46c559cMerge tag '9.2.beta9' into t/27754/public/python3.8

comment:191 Changed 2 months ago by mkoeppe

  • Status changed from needs_review to positive_review

comment:192 Changed 2 months ago by vbraun

  • Status changed from positive_review to needs_work
    STDOUT: CONFLICT (content): Merge conflict in src/sage/graphs/generic_graph.py

comment:193 Changed 2 months ago by git

  • Commit changed from 46c559c0b052e37d87c050fee879150318ba0463 to a625ed34796120b6d8bc7397c4a9ef4314649332

Branch pushed to git repo; I updated commit sha1. New commits:

f45221btrac #30140: remove deprecated method ste_planar_positions
a625ed3Merge branch 'public/graphs/30140_remove_set_plananr_positions' of git://trac.sagemath.org/sage into t/27754/public/python3.8

comment:194 Changed 2 months ago by git

  • Commit changed from a625ed34796120b6d8bc7397c4a9ef4314649332 to 53b569d68ddca39a0c5b33ab59ce77dd651da21a

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

3db87adbuild/pkgs/maxima: Update to 5.44.0
bcf3c98Remove old patches
07aa2f9build/pkgs/maxima/spkg-install.in: New workaround for makeinfo trouble
046634dsage.interfaces.maxima_abstract.MaximaAbstract._command_runner: Skip all comment lines, rather than counting lines to skip
d1ad8d7sage.interfaces.maxima_abstract.MaximaAbstract.completions: Remove '-impl' functions
2ea3e73Merge branch 'u/mkoeppe/upgrade_maxima_to_5_44_0' of trac.sagemath.org:sage into HEAD
8f83f96#30063 : Maxima gamma_greek is now called gamma_incomplete_lower
21e2fea#30063 : mark Maxima's integration regression as known bug.
d6e0952Trac #30063: Upgrade Maxima to 5.44.0
53b569dMerge remote-tracking branch 'vbraun/develop' into t/27754/public/python3.8

comment:195 Changed 2 months ago by mkoeppe

  • Dependencies changed from #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365 to #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365, #30140
  • Status changed from needs_work to positive_review

comment:196 Changed 2 months ago by vbraun

  • Status changed from positive_review to needs_work

I'm getting various failures on 32-bit apparently due to different iterator order:

**********************************************************************
File "src/sage/combinat/rigged_configurations/tensor_product_kr_tableaux.py", line 333, in sage.combinat.rigged_configurations.tensor_product_kr_tableaux.TensorProductOfKirillovReshetikhinTableaux.__iter__
Failed example:
    next(g)
Expected:
    [[2], [4]] (X) [[1]]
Got:
    [[1], [2]] (X) [[1]]
**********************************************************************
1 item had failures:
   1 of   5 in sage.combinat.rigged_configurations.tensor_product_kr_tableaux.TensorProductOfKirillovReshetikhinTableaux.__iter__
    [95 tests, 1 failure, 21.93 s]
**********************************************************************
File "src/sage/geometry/cone.py", line 4059, in sage.geometry.cone.?.semigroup_generators
Failed example:
    halfplane.semigroup_generators()
Expected:
    (N(0, 1), N(1, 0), N(-1, 0))
Got:
    (N(-1, 0), N(1, 0), N(0, 1))
**********************************************************************
1 item had failures:
   1 of  23 in sage.geometry.cone.?.semigroup_generators
    [1241 tests, 1 failure, 18.70 s]
**********************************************************************
File "src/sage/geometry/fan_isomorphism.py", line 124, in sage.geometry.fan_isomorphism.fan_isomorphism_generator
Failed example:
    tuple(fan_isomorphism_generator(fan1, fan2))
Expected:
    (
    [18  1 -5]  [ 6 -3  7]
    [ 4  0 -1]  [ 1 -1  2]
    [ 5  0 -1], [ 2 -1  2]
    )
Got:
    (
    [ 6 -3  7]  [18  1 -5]
    [ 1 -1  2]  [ 4  0 -1]
    [ 2 -1  2], [ 5  0 -1]
    )
**********************************************************************
File "src/sage/geometry/fan_isomorphism.py", line 130, in sage.geometry.fan_isomorphism.fan_isomorphism_generator
Failed example:
    tuple(fan_isomorphism_generator(fan2, fan1))
Expected:
    (
    [ 0 -1  1]  [ 0 -1  1]
    [ 1 -7  2]  [ 2 -2 -5]
    [ 0 -5  4], [ 1  0 -3]
    )
Got:
    (
    [ 0 -1  1]  [ 0 -1  1]
    [ 2 -2 -5]  [ 1 -7  2]
    [ 1  0 -3], [ 0 -5  4]
    )
**********************************************************************
1 item had failures:
   2 of  21 in sage.geometry.fan_isomorphism.fan_isomorphism_generator
    [65 tests, 2 failures, 1.07 s]
**********************************************************************
File "src/sage/geometry/polyhedron/library.py", line 2639, in sage.geometry.polyhedron.library.Polytopes.generalized_permutahedron
Failed example:
    sorted(perm_a2_reg.vertices())
Expected:
    [A vertex at (-1, 0),
     A vertex at (-1/2, -0.866025403784439?),
     A vertex at (-1/2, 0.866025403784439?),
     A vertex at (1/2, -0.866025403784439?),
     A vertex at (1/2, 0.866025403784439?),
     A vertex at (1.000000000000000?, 0.?e-18)]
Got:
    [A vertex at (-1.000000000000000?, 0.?e-18),
     A vertex at (-1/2, -0.866025403784439?),
     A vertex at (-1/2, 0.866025403784439?),
     A vertex at (1/2, -0.866025403784439?),
     A vertex at (1/2, 0.866025403784439?),
     A vertex at (1, 0)]
**********************************************************************
1 item had failures:
   1 of  21 in sage.geometry.polyhedron.library.Polytopes.generalized_permutahedron
    [305 tests, 1 failure, 97.96 s]
**********************************************************************
File "src/sage/geometry/triangulation/point_configuration.py", line 1401, in sage.geometry.triangulation.point_configuration.PointConfiguration.circuits_support
Failed example:
    list( p.circuits_support() )
Expected:
    [(0, 3, 4), (0, 1, 2), (1, 2, 3, 4)]
Got:
    [(0, 1, 2), (0, 3, 4), (1, 2, 3, 4)]
**********************************************************************
File "src/sage/geometry/triangulation/point_configuration.py", line 1471, in sage.geometry.triangulation.point_configuration.PointConfiguration.circuits
Failed example:
    p.circuits()
Expected:
    (((0,), (1, 2), (3, 4)), ((0,), (3, 4), (1, 2)), ((1, 2), (0,), (3, 4)))
Got:
    (((0,), (3, 4), (1, 2)), ((0,), (1, 2), (3, 4)), ((1, 2), (0,), (3, 4)))
**********************************************************************
2 items had failures:
   1 of   6 in sage.geometry.triangulation.point_configuration.PointConfiguration.circuits
   1 of   3 in sage.geometry.triangulation.point_configuration.PointConfiguration.circuits_support
    [214 tests, 2 failures, 19.74 s]
**********************************************************************
File "src/sage/graphs/graph.py", line 7342, in sage.graphs.graph.Graph.cliques_containing_vertex
Failed example:
    sorted(d[(0, 1)])
Expected:
    [[(0, 1), (0, 0)], [(0, 1), (0, 2)], [(0, 1), (1, 1)]]
Got:
    [[(0, 0), (0, 1)], [(0, 2), (0, 1)], [(1, 1), (0, 1)]]
**********************************************************************
1 item had failures:
   1 of  15 in sage.graphs.graph.Graph.cliques_containing_vertex
    [1190 tests, 1 failure, 20.84 s]
**********************************************************************
File "src/sage/homology/homology_vector_space_with_basis.py", line 113, in sage.homology.homology_vector_space_with_basis.HomologyVectorSpaceWithBasis
Failed example:
    x.cup_product(x)
Expected:
    0
Got:
    h^{2,0}
**********************************************************************
File "src/sage/homology/homology_vector_space_with_basis.py", line 115, in sage.homology.homology_vector_space_with_basis.HomologyVectorSpaceWithBasis
Failed example:
    x.cup_product(y)
Expected:
    h^{2,0}
Got:
    0
**********************************************************************
1 item had failures:
   2 of  45 in sage.homology.homology_vector_space_with_basis.HomologyVectorSpaceWithBasis
    [194 tests, 2 failures, 16.93 s]
R[write to console]: Warning messages:
R[write to console]: 1: 
R[write to console]: In sage10 + sage6 :
R[write to console]: 
 
R[write to console]:  longer object length is not a multiple of shorter object length
R[write to console]: 2: 
R[write to console]: In sqrt(sage10) :
R[write to console]:  NaNs produced
R[write to console]: 3: 
R[write to console]: In sqrt(sage4) :
R[write to console]:  NaNs produced
**********************************************************************
File "src/sage/matroids/graphic_matroid.py", line 1659, in sage.matroids.graphic_matroid.GraphicMatroid.graphic_coextensions
Failed example:
    for N1 in I:
        N1.graph().edges(sort=True)
Expected:
    [(0, 1, 0), (0, 3, 1), (0, 4, 'a'), (1, 2, 2), (2, 3, 3)]
    [(0, 1, 0), (0, 3, 1), (1, 4, 2), (2, 3, 3), (2, 4, 'a')]
Got:
    [(0, 1, 0), (0, 3, 1), (0, 4, 'a'), (1, 2, 2), (2, 3, 3)]
    [(0, 3, 1), (0, 4, 0), (1, 2, 2), (1, 4, 'a'), (2, 3, 3)]
**********************************************************************
1 item had failures:
   1 of  27 in sage.matroids.graphic_matroid.GraphicMatroid.graphic_coextensions
    [358 tests, 1 failure, 0.75 s]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py", line 1099, in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.class_representatives
Failed example:
    R
Expected:
    [[V(2)*V(3)], [V(1)*V(2)]]
Got:
    [[V(1)*V(2)], [V(2)*V(3)]]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py", line 1101, in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.class_representatives
Failed example:
    [v.continued_fraction()[1] for v in R]
Expected:
    [(1, 2, 2), (3,)]
Got:
    [(3,), (1, 2, 2)]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py", line 1318, in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.reduced_elements
Failed example:
    R
Expected:
    [
    [ 5*lam     -3]  [ 5*lam     -7]  [4*lam    -3]  [3*lam    -1]
    [     7 -2*lam], [     3 -2*lam], [    3  -lam], [    1     0]
    ]
Got:
    [
    [3*lam    -1]  [ 5*lam     -3]  [ 5*lam     -7]  [4*lam    -3]
    [    1     0], [     7 -2*lam], [     3 -2*lam], [    3  -lam]
    ]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py", line 1323, in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.reduced_elements
Failed example:
    [v.continued_fraction() for v in R]
Expected:
    [((), (1, 2, 2)), ((), (2, 2, 1)), ((), (2, 1, 2)), ((), (3,))]
Got:
    [((), (3,)), ((), (1, 2, 2)), ((), (2, 2, 1)), ((), (2, 1, 2))]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py", line 1350, in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.simple_elements
Failed example:
    G.simple_elements(D=14)
Expected:
    [
    [2*lam     1]  [  lam     1]  [2*lam     3]  [  lam     3]
    [    3   lam], [    3 2*lam], [    1   lam], [    1 2*lam]
    ]
Got:
    [
    [2*lam     3]  [  lam     3]  [2*lam     1]  [  lam     1]
    [    1   lam], [    1 2*lam], [    3   lam], [    3 2*lam]
    ]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py", line 1387, in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.rational_period_functions
Failed example:
    G.rational_period_functions(k=-4, D=14)
Expected:
    [-z^4 + 1, 16*z^4 - 16, -16*z^4 + 16]
Got:
    [-z^4 + 1, -16*z^4 + 16, 16*z^4 - 16]
**********************************************************************
4 items had failures:
   2 of  27 in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.class_representatives
   1 of   7 in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.rational_period_functions
   2 of   9 in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.reduced_elements
   1 of   5 in sage.modular.modform_hecketriangle.hecke_triangle_groups.?.simple_elements
    [228 tests, 6 failures, 4.90 s]
**********************************************************************
File "src/sage/modular/modform_hecketriangle/readme.py", line 549, in sage.modular.modform_hecketriangle.readme
Failed example:
    G.class_representatives(9*G.lam() + 5)
Expected:
    [S*T^(-2)*S*T^(-1)*S, T*S*T^2]
Got:
    [T*S*T^2, S*T^(-2)*S*T^(-1)*S]
**********************************************************************
1 item had failures:
   1 of 383 in sage.modular.modform_hecketriangle.readme
    [382 tests, 1 failure, 19.56 s]
**********************************************************************
File "src/sage/monoids/trace_monoid.py", line 959, in sage.monoids.trace_monoid.TraceMonoid._repr_
Failed example:
    M.<a,b,c,d> = TraceMonoid(I=I); M
Expected:
    Trace monoid on 4 generators ([a], [b], [c], [d])
     with independence relation {{a, d}, {b, c}}
Got:
    Trace monoid on 4 generators ([a], [b], [c], [d]) with independence relation {{b, c}, {d, a}}
**********************************************************************
File "src/sage/monoids/trace_monoid.py", line 976, in sage.monoids.trace_monoid.TraceMonoid._latex_
Failed example:
    M.<a,b,c,d> = TraceMonoid(I=I); latex(M)
Expected:
    \langle a, b, c, d \mid ad=da,bc=cb \rangle
Got:
    \langle a, b, c, d \mid bc=cb,da=ad \rangle
**********************************************************************
2 items had failures:
   1 of   4 in sage.monoids.trace_monoid.TraceMonoid._latex_
   1 of   4 in sage.monoids.trace_monoid.TraceMonoid._repr_
    [186 tests, 2 failures, 4.63 s]
**********************************************************************
File "src/sage/rings/polynomial/multi_polynomial_ideal.py", line 4166, in sage.rings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_basis
Failed example:
    f
Expected:
    1/2*y^2*z^7 - 1/4*y*z^8 + 2*x*z^5 + 95*z^6 + 1/2*y^5 - 1/4*y^4*z + x^2*y^2 + 3/2*x^2*y*z + 95*x*y*z^2
Got:
    1/2*y^2*z^7 - 1/4*y*z^8 + 95*x*z^5 + 2*z^6 + 1/2*y^5 - 1/4*y^4*z + x^2*y^2 + 189/2*x^2*y*z + 2*x*y*z^2
**********************************************************************
File "src/sage/rings/polynomial/multi_polynomial_ideal.py", line 4168, in sage.rings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_basis
Failed example:
    f.lift(I.gens())
Expected:
    [2*x + 95*z, 1/2*y^2 - 1/4*y*z, 0]
Got:
    [95*x + 2*z, 1/2*y^2 - 1/4*y*z, 0]
**********************************************************************
File "src/sage/rings/polynomial/multi_polynomial_ideal.py", line 4170, in sage.rings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_basis
Failed example:
    l = f.lift(J.gens()); l
Expected:
    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1/2*y^2 + 1/4*y*z, 1/2*y^2*z^2 - 1/4*y*z^3 + 2*x + 95*z]
Got:
    [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, -1/2*y^2 + 1/4*y*z, 1/2*y^2*z^2 - 1/4*y*z^3 + 95*x + 2*z]
**********************************************************************
1 item had failures:
   3 of 111 in sage.rings.polynomial.multi_polynomial_ideal.NCPolynomialIdeal.groebner_basis
    [917 tests, 3 failures, 10.04 s]
----------------------------------------------------------------------
sage -t --long --random-seed=0 src/sage/combinat/rigged_configurations/tensor_product_kr_tableaux.py  # 1 doctest failed
sage -t --long --random-seed=0 src/sage/geometry/cone.py  # 1 doctest failed
sage -t --long --random-seed=0 src/sage/geometry/fan_isomorphism.py  # 2 doctests failed
sage -t --long --random-seed=0 src/sage/geometry/polyhedron/library.py  # 1 doctest failed
sage -t --long --random-seed=0 src/sage/geometry/triangulation/point_configuration.py  # 2 doctests failed
sage -t --long --random-seed=0 src/sage/graphs/graph.py  # 1 doctest failed
sage -t --long --random-seed=0 src/sage/homology/homology_vector_space_with_basis.py  # 2 doctests failed
sage -t --long --random-seed=0 src/sage/matroids/graphic_matroid.py  # 1 doctest failed
sage -t --long --random-seed=0 src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py  # 6 doctests failed
sage -t --long --random-seed=0 src/sage/modular/modform_hecketriangle/readme.py  # 1 doctest failed
sage -t --long --random-seed=0 src/sage/monoids/trace_monoid.py  # 2 doctests failed
sage -t --long --random-seed=0 src/sage/rings/polynomial/multi_polynomial_ideal.py  # 3 doctests failed
----------------------------------------------------------------------

comment:197 Changed 2 months ago by git

  • Commit changed from 53b569d68ddca39a0c5b33ab59ce77dd651da21a to 42bfb1f7a85fdad131bd9b278105641952a54bfb

Branch pushed to git repo; I updated commit sha1. New commits:

42bfb1fMerge tag '9.2.beta10' into t/27754/public/python3.8

comment:198 Changed 2 months ago by git

  • Commit changed from 42bfb1f7a85fdad131bd9b278105641952a54bfb to d167e08bb10a7c0b36be21edeffd75e8b222d89b

Branch pushed to git repo; I updated commit sha1. New commits:

d167e08Make doctests more flexible for Python 3.8 on 32bit

comment:199 Changed 2 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:200 Changed 2 months ago by mkoeppe

This fixes all failures except for the following two:

sage -t --random-seed=0 src/sage/homology/homology_vector_space_with_basis.py  # 2 doctests failed
sage -t --random-seed=0 src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py  # 1 doctest failed

Help with these would be welcome

comment:201 Changed 2 months ago by jhpalmieri

There must be some way for me to run a 32-bit machine via Docker so I can look into these, but I can't figure out how.

comment:202 follow-up: Changed 2 months ago by mkoeppe

Yes, for example EXTRA_CONFIGURE_ARGS="--without-system-suitesparse" tox -e docker-ubuntu-eoan-i386-standard-python3_spkg

Last edited 2 months ago by mkoeppe (previous) (diff)

comment:203 in reply to: ↑ 202 Changed 2 months ago by jhpalmieri

Replying to mkoeppe:

Yes, for example EXTRA_CONFIGURE_ARGS="--without-system-suitesparse" tox -e docker-ubuntu-eoan-i386-standard-python3_spkg

Well, that didn't work for me (with EXTRA_CONFIGURE_ARGS="--with-system-suitesparse=no"): the build failed for cypari and giac. But I used VirtualBox? to install a 32-bit Debian machine, and I see the doctest failures. I actually haven't looked at the hecke_triangle_groups.py failure — not my area of expertise — but I understand what's happening with the homology failure. There are some unsorted pieces of data lying around, and when they get converted to lists or tuples, how they get converted affects the answer. Both answers (32-bit and 64-bit) are mathematically correct, it's just that you get a different choice of bases in the two cases.

I propose changing to a case where the sorting won't affect the answer.

comment:204 Changed 2 months ago by git

  • Commit changed from d167e08bb10a7c0b36be21edeffd75e8b222d89b to d85c4ba66d5bf4b03e4f480da0ca8d49ade49fb2

Branch pushed to git repo; I updated commit sha1. New commits:

d85c4batrac 27754: homology doctest fix

comment:205 follow-up: Changed 2 months ago by jhpalmieri

For the hecke_triangle_groups.py problem, this seems to fix it, but I don't know the mathematics of what's going on, nor have I looked at the code:

  • src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py

    diff --git a/src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py b/src/sage/modular/modform_hecketriangle/hecke_triangle_groups.py
    index 47ba82eb8e..cb336f64c4 100644
    a b class HeckeTriangleGroup(FinitelyGeneratedMatrixGroup_generic, 
    10961096            [[U], [U^(-1)]]
    10971097
    10981098            sage: R = G.class_representatives(14)
    1099             sage: R
     1099            sage: sorted(R)
    11001100            [[V(2)*V(3)], [V(1)*V(2)]]
    11011101            sage: sorted(v.continued_fraction()[1] for v in R)
    11021102            [(1, 2, 2), (3,)]

comment:206 in reply to: ↑ 205 ; follow-up: Changed 2 months ago by dimpase

Replying to jhpalmieri:

For the hecke_triangle_groups.py problem, this seems to fix it, but I don't know the mathematics of what's going on, nor have I looked at the code:

docs say that it's a list of representatives, with no meaning attached to the order, so your fix is OK.

comment:207 in reply to: ↑ 206 Changed 2 months ago by jhpalmieri

Replying to dimpase:

Replying to jhpalmieri:

For the hecke_triangle_groups.py problem, this seems to fix it, but I don't know the mathematics of what's going on, nor have I looked at the code:

docs say that it's a list of representatives, with no meaning attached to the order, so your fix is OK.

I'll push a branch after I do a bit more testing, unless someone beats me to it. (One question is how the elements in the list R are sorted. I can't tell after looking at the code for a few minutes. But this does seem to work.)

comment:208 Changed 2 months ago by git

  • Commit changed from d85c4ba66d5bf4b03e4f480da0ca8d49ade49fb2 to b9d451c7740613f38194cba3c4bd6f428cdc19e5

Branch pushed to git repo; I updated commit sha1. New commits:

b9d451ctrac 27754: 32-bit Python 3.8 doctest fix in hecke_triangle_groups.py

comment:209 Changed 2 months ago by jhpalmieri

Okay, this works for me on OS X with the system Python and Sage's Python 3.8, and the same on a virtual Debian 32-bit machine.

comment:210 Changed 2 months ago by mkoeppe

  • Status changed from needs_review to positive_review

Great!

comment:211 Changed 2 months ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:212 Changed 2 months ago by git

  • Commit changed from b9d451c7740613f38194cba3c4bd6f428cdc19e5 to a9b69650667022f4a46b59ac90994254389931c1

Branch pushed to git repo; I updated commit sha1. New commits:

488bba4on macOS set OMP_NUM_THREADS to 1 for docbuild and tests
4b11a70use the correct import location, not six.*
a9b6965Merge commit '4b11a706e1267b385099d8691b4468efd974f94d' of git://trac.sagemath.org/sage into t/27754/public/python3.8

comment:213 Changed 2 months ago by mkoeppe

  • Status changed from needs_work to positive_review

comment:214 Changed 2 months ago by mkoeppe

  • Dependencies changed from #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365, #30140 to #28197, #29042, #29861, #30162, #30225, #30127, #30144, #30345, #30365, #30140, #30351

comment:215 Changed 2 months ago by git

  • Commit changed from a9b69650667022f4a46b59ac90994254389931c1 to f1cda120e6c715ec6bdd6d8bb776b3b0aa921794
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

f1cda12Fixup merge

comment:216 Changed 2 months ago by mkoeppe

  • Status changed from needs_review to positive_review

comment:217 Changed 8 weeks ago by mkoeppe

  • Priority changed from critical to blocker

comment:218 Changed 8 weeks ago by charpent

FWIW, worked for MeToo(TM) :

Rebuilding 2.9.beta11+#27754 on Debian sesting running on core i7 + 16 GB RAM gives results identical to those obtained on the same machine without #27754 (see sage-release). And $sSAGE_LOCAL\bin\python* are symlinks to \usr\bin\...

Last edited 8 weeks ago by charpent (previous) (diff)

comment:219 Changed 8 weeks ago by dimpase

it even works on Rasberry Pi 4 (armv7l).

comment:220 Changed 7 weeks ago by vbraun

  • Branch changed from public/python3.8 to f1cda120e6c715ec6bdd6d8bb776b3b0aa921794
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:221 Changed 7 weeks ago by slelievre

  • Commit f1cda120e6c715ec6bdd6d8bb776b3b0aa921794 deleted

Wonderful to have this in Sage 9.2.

Next episode:

(Python 3.9.rc2 is expected on 2020-09-14 and Python 3.9 final on 2020-10-05.)

Note: See TracTickets for help on using tickets.