Opened 16 months ago

Closed 7 months ago

Last modified 6 months ago

#25680 closed enhancement (fixed)

Upgrade to Python 3.7.x

Reported by: jdemeyer Owned by:
Priority: major Milestone: sage-8.8
Component: packages: standard Keywords: upgrade, Python3
Cc: chapoton, embray, fbissey, mkoeppe, slelievre Merged in:
Authors: Jeroen Demeyer Reviewers: Vincent Klein, Erik Bray, Frédéric Chapoton
Report Upstream: N/A Work issues:
Branch: 5a09cf1 (Commits) Commit:
Dependencies: Stopgaps:

Attachments (2)

python3-3.7.0.log (197.1 KB) - added by vklein 16 months ago.
matrix1.c.bz2 (88.9 KB) - added by fbissey 6 months ago.

Download all attachments as: .zip

Change History (193)

comment:1 Changed 16 months ago by jdemeyer

  • Branch set to u/jdemeyer/upgrade_to_python_3_7_0

comment:2 Changed 16 months ago by jdemeyer

  • Authors set to Jeroen Demeyer
  • Commit set to 2c67605bfb65c6d52b0697bbe0a027afb7172d4e
  • Status changed from new to needs_review

New commits:

2c67605Upgrade to Python 3.7.0

comment:3 follow-ups: Changed 16 months ago by embray

I don't think we should be in any hurry on this. All the last ~year's work of Python 3 porting has been targeting Python 3.6, and I think I'd rather stick with that before risking throwing everything out of whack again.

Between Python 3.6 and 3.7 I suspect the differences, if any, will be minor. But I'd rather try to stabilize on 3.6 first (to which we're quite close), then deal with those differences, rather than change Python versions again. Also I don't know that Python 3.7 has been ported to Cygwin yet (although it does contain a number of useful fixes for Cygwin, I do know).

That said, if you know any specific changes in Python 3.7 (other than the aforementioned Cygwin fixes) that would actively make our porting effort easier I'd consider it...

comment:4 Changed 16 months ago by embray

  • Status changed from needs_review to needs_info

comment:5 follow-up: Changed 16 months ago by chapoton

Jeroen has said elsewhere that islice will accept Sage integers instead of just ints.

I am again becoming tired and frustated by the slow pace of progress towards py3. And by the large amount of difficult work that remains..

comment:6 Changed 16 months ago by embray

Ah, I remember making an issue about that, but I don't think I followed what the resolution was.

comment:7 Changed 16 months ago by git

  • Commit changed from 2c67605bfb65c6d52b0697bbe0a027afb7172d4e to 1bfcdc84ccc6bc18d36b2dd15a3daa0ae2e52ad2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

1bfcdc8Upgrade to Python 3.7.0

comment:8 Changed 16 months ago by embray

Well there's this https://github.com/python/cpython/pull/1918 But unless there are other fixes I'm not seeing it's annoying that this was only fixed for islice, where I believe there is other code affected by the same issue (e.g. range). Also we've already worked aroudn the majority of those cases (sigh...)

Last edited 16 months ago by embray (previous) (diff)

comment:9 in reply to: ↑ 5 Changed 16 months ago by embray

Replying to chapoton:

I am again becoming tired and frustated by the slow pace of progress towards py3. And by the large amount of difficult work that remains..

Unfortunately I'm not convinced that upgrading Python yet again is going to make that go any faster. In any case I don't actually have that much "difficult" work remaining. Just dozens of little issues. My Python 3 branch has < 600 modules failing, and most of it appears to minor issues (at one point I had it down to ~400 and I don't know why it ballooned up again, but one or two small changes in just the right places can do that with Sage...)

Also I'll add if you're not using a patched pynac you're going to have lots, lots more "serious" looking issues. We still need to get a pynac upgrade in (I've been manually installing pynac with my Python 3 fixes every time I re-build sage).

comment:10 in reply to: ↑ 3 Changed 16 months ago by jdemeyer

Replying to embray:

That said, if you know any specific changes in Python 3.7 (other than the aforementioned Cygwin fixes) that would actively make our porting effort easier I'd consider it...

#25391 is a pretty serious issue which is fixed by a Python upgrade (probably also by upgrading to 3.6.6 but I haven't tested that).

comment:11 in reply to: ↑ 3 Changed 16 months ago by jdemeyer

Replying to embray:

I don't think we should be in any hurry on this. All the last ~year's work of Python 3 porting has been targeting Python 3.6, and I think I'd rather stick with that before risking throwing everything out of whack again.

First of all, I do not think that this 3.6 -> 3.7 will throw everything out of whack.

However, regardless of that, we'll have to upgrade sooner or later. I don't see the point of having a perfectly working Sage on an outdated Python version. And for the many issues that we still have to fix, it would be better to make sure that they work on Python 3.7 from the start.

Last edited 16 months ago by jdemeyer (previous) (diff)

comment:12 follow-up: Changed 16 months ago by chapoton

I still think we have very hard remaining problems, among which

  • sorting issues in graphs
  • hashing and comparisons, especially for groups
  • coercion framework have broken parts

comment:13 Changed 16 months ago by jdemeyer

  • Cc fbissey mkoeppe added

Changed 16 months ago by vklein

comment:14 Changed 16 months ago by vklein

I get the following error

$ ./configure --with-python=3
...
$ make build
...
[python3-3.7.0] Testing importing of various modules...
[python3-3.7.0] Traceback (most recent call last):
[python3-3.7.0]   File "<string>", line 1, in <module>
[python3-3.7.0]   File "/home/vklein/odk/sage/local/var/tmp/sage/build/python3-3.7.0/src/Lib/ctypes/__init__.py", line 7, in <module>
[python3-3.7.0]     from _ctypes import Union, Structure, Array
[python3-3.7.0] ModuleNotFoundError: No module named '_ctypes'
[python3-3.7.0] ctypes module failed to import
[python3-3.7.0] math module imported OK
[python3-3.7.0] hashlib module imported OK
[python3-3.7.0] crypt module imported OK
[python3-3.7.0] readline module imported OK
[python3-3.7.0] socket module imported OK
[python3-3.7.0] Error: One or more modules failed to import.
...

full python3.7 install log in attachement

Last edited 16 months ago by vklein (previous) (diff)

comment:15 follow-up: Changed 16 months ago by fbissey

Failure to build _ctypes usually means a problem with ffi (sometimes called libffi) may be there is minimum version requirement?

comment:16 Changed 16 months ago by vklein

Indeed i don't get the error after installing libffi-dev. Thanks !

Note : sage make build works fine in python 3.6 without that.

Last edited 16 months ago by vklein (previous) (diff)

comment:17 Changed 16 months ago by vklein

next error during the build is :

Installing collected packages: ipykernel
  Running setup.py install for ipykernel: started
    Running command /home/vklein/odk/sage/local/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-qoxsao1r-build/setup.py';f=getattr(tokenize
, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --no-user-cfg install --record /tmp/pip-vc23lh
pc-record/install-record.txt --single-version-externally-managed --compile
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-qoxsao1r-build/setup.py", line 91, in <module>
        from ipykernel.kernelspec import write_kernel_spec, make_ipkernel_cmd, KERNEL_NAME
...
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/pexpect/spawnbase.py", line 224
        def expect(self, pattern, timeout=-1, searchwindowsize=-1, async=False):
                                                                       ^
    SyntaxError: invalid syntax
    Running setup.py install for ipykernel: finished with status 'error'
Cleaning up...

Issue reference : https://github.com/pypa/pipenv/issues/956

comment:18 Changed 16 months ago by fbissey

To be honest we haven't updated pexpect in vanilla sage for a while. It may be a good idea to do that - in a separate ticket.

comment:19 Changed 16 months ago by vklein

Do we need to update pexpect or to patch pipenv ?

comment:20 Changed 16 months ago by jdemeyer

No, pexpect.

comment:21 follow-up: Changed 16 months ago by fbissey

I am at pexpect-4.2.1 in sage-on-gentoo (without any patches) and sage is at 4.1.0 + some patches.

comment:22 in reply to: ↑ 21 Changed 16 months ago by jdemeyer

Replying to fbissey:

I am at pexpect-4.2.1 in sage-on-gentoo (without any patches) and sage is at 4.1.0 + some patches.

The patches in Sage are for performance. Without those patches, pexpect is really a lot slower.

comment:23 Changed 16 months ago by vklein

  • Dependencies set to #25700

Let's do that #25700.

comment:24 in reply to: ↑ 12 ; follow-up: Changed 16 months ago by embray

Replying to chapoton:

I still think we have very hard remaining problems, among which

  • sorting issues in graphs
  • hashing and comparisons, especially for groups
  • coercion framework have broken parts

Most of which Python 3.7 won't help with. The one area where it might is sorting of dicts, since dict insertion order is now preserved (hooray!)

Well, I'm not completely opposed to upgrading, just skeptical, and don't want to rush into it. When I get a chance I'll compare how my python3 branch fares with and without it.

comment:25 in reply to: ↑ 24 Changed 16 months ago by vklein

Replying to embray:

Well, I'm not completely opposed to upgrading, just skeptical, and don't want to rush into it. When I get a chance I'll compare how my python3 branch fares with and without it.

One of my goal reviewing this ticket is to produce a kind of diff between the doctests failures in 3.6 and 3.7. It can be helpful to decide what to do.

comment:26 follow-up: Changed 16 months ago by vklein

And now it's numpy's turn :

$ ./configure --with-python=3
...
$ make build
...
/home/vklein/odk/sage/local/lib/python3.7/distutils/dist.py:274: UserWarning: Un
known distribution option: 'define_macros'
  warnings.warn(msg)
numpy/random/mtrand/mtrand.c: In function '__Pyx__ExceptionSave':
numpy/random/mtrand/mtrand.c:45208:19: error: 'PyThreadState {aka struct _ts}' h
as no member named 'exc_type'
     *type = tstate->exc_type;
                   ^
numpy/random/mtrand/mtrand.c:45209:20: error: 'PyThreadState {aka struct _ts}' h
as no member named 'exc_value'
     *value = tstate->exc_value;
...

numpy's issue #10500. I am not sure what is the right call here i will try with numpy 1.14.5

comment:27 Changed 16 months ago by vklein

When updating to numpy 1.14.5 with this following patch :

diff --git a/numpy/distutils/system_info.py b/numpy/distutils/system_info.py
index 93a8e6f..6e31d30 100644
--- a/numpy/distutils/system_info.py
+++ b/numpy/distutils/system_info.py
@@ -1740,7 +1740,7 @@ class blas_info(system_info):
             lib = self.has_cblas(info)
             if lib is not None:
                 info['language'] = 'c'
-                info['libraries'] = [lib]
+                info['libraries'] = lib
                 info['define_macros'] = [('HAVE_CBLAS', None)]
         self.set_info(**info)
 
@@ -1772,16 +1772,16 @@ class blas_info(system_info):
                 # check for cblas lib, and if not present check for blas lib.
                 try:
                     c.link_executable(obj, os.path.join(tmpdir, "a.out"),
-                                      libraries=["cblas"],
+                                      libraries=["libraries"],
                                       library_dirs=info['library_dirs'],
                                       extra_postargs=info.get('extra_link_args', []))
-                    res = "cblas"
+                    res = info["libraries"]
                 except distutils.ccompiler.LinkError:
                     c.link_executable(obj, os.path.join(tmpdir, "a.out"),
                                       libraries=["blas"],
                                       library_dirs=info['library_dirs'],
                                       extra_postargs=info.get('extra_link_args', []))
-                    res = "blas"
+                    res = ["blas"]
             except distutils.ccompiler.CompileError:
                 res = None
         finally:

I get the following error :

Found local metadata for numpy-1.14.5.p0
Using cached file /home/vklein/odk/sage/upstream/numpy-1.14.5.zip
numpy-1.14.5.p0
====================================================
Setting up build directory for numpy-1.14.5.p0
Finished extraction
Applying patches from ../patches...
Applying ../patches/numpy-1.14.5-no-hardcode-blas.patch
patching file numpy/distutils/system_info.py
****************************************************
Host system:
Linux tuono 4.13.0-45-generic #50~16.04.1-Ubuntu SMP Wed May 30 11:18:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.10' --with-bugurl=file:///usr/s
hare/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suff
ix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=po
six --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcx
x-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --en
able-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/
usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 -
-with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share
/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --wit
h-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-g
nu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 
****************************************************

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last NumPy release on PyPi)


blas_opt_info:
blas_mkl_info:
Disabled blas_mkl_info: (MKLROOT is None)
Disabled blas_mkl_info: (MKLROOT is None)
customize UnixCCompiler
  libraries mkl_rt not found in []
  NOT AVAILABLE

blis_info:
customize UnixCCompiler
  libraries blis not found in ['/home/vklein/odk/sage/local/lib', '/usr/local/lib', '/usr/lib', '/usr/lib/x86_64-li
nux-gnu']
  NOT AVAILABLE

openblas_info:
customize UnixCCompiler
Disabled openblas_info: (OPENBLAS is None)
customize UnixCCompiler
  libraries openblas not found in []
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Disabled atlas_3_10_blas_threads_info: (PTATLAS is None)
customize UnixCCompiler
  libraries tatlas not found in []
  NOT AVAILABLE

atlas_3_10_blas_info:
Disabled atlas_3_10_blas_info: (ATLAS is None)
customize UnixCCompiler
  libraries satlas not found in []
  NOT AVAILABLE

atlas_blas_threads_info:
Disabled atlas_blas_threads_info: (PTATLAS is None)
customize UnixCCompiler
  libraries ptf77blas,ptcblas,atlas not found in []
  NOT AVAILABLE

atlas_blas_info:
Disabled atlas_blas_info: (ATLAS is None)
customize UnixCCompiler
  libraries f77blas,cblas,atlas not found in []
  NOT AVAILABLE

blas_info:
customize UnixCCompiler
customize UnixCCompiler
C compiler: gcc -DNDEBUG -g -fwrapv -O3 -Wall -D__CEPHES_COMPLEX_H -fPIC

creating /tmp/tmpyq7p80or/tmp
creating /tmp/tmpyq7p80or/tmp/tmpyq7p80or
compile options: '-I/home/vklein/odk/sage/local/include -c'
gcc: /tmp/tmpyq7p80or/source.c
gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/lib -llibraries -o /tmp/tmpyq7p80or/a.o
ut
/usr/bin/ld: cannot find -llibraries
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -llibraries
collect2: error: ld returned 1 exit status
gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/lib -lblas -o /tmp/tmpyq7p80or/a.out
Running from numpy source directory.
/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/system_info.py:624: UserWarning:
 
    Atlas (http://math-atlas.sourceforge.net/) libraries not found.
    Directories to search for the libraries can be specified in the
    numpy/distutils/site.cfg file (section [atlas]) or by setting
    the ATLAS environment variable.
  self.calc_info()
/usr/bin/ld: cannot find -lblas
collect2: error: ld returned 1 exit status
/usr/bin/ld: cannot find -lblas
collect2: error: ld returned 1 exit status
Traceback (most recent call last):
  File "/home/vklein/odk/sage/local/lib/python3.7/distutils/unixccompiler.py", line 197, in link
    self.spawn(linker + ld_args)
  File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/ccompiler.py", line 89, 
in <lambda>
    m = lambda self, *args, **kw: func(self, *args, **kw)
  File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/ccompiler.py", line 152,
 in CCompiler_spawn
    raise DistutilsExecError('Command "%s" failed with exit status %d%s' % (cmd, s, msg))
distutils.errors.DistutilsExecError: Command "gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage
/local/lib -llibraries -o /tmp/tmpyq7p80or/a.out" failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vklein/odk/sage/local/var/tmp/sage/build/numpy-1.14.5.p0/src/numpy/distutils/system_info.py", line 17
77, in has_cblas
    extra_postargs=info.get('extra_link_args', []))
  File "/home/vklein/odk/sage/local/lib/python3.7/distutils/ccompiler.py", line 734, in link_executable
    debug, extra_preargs, extra_postargs, None, target_lang)
  File "/home/vklein/odk/sage/local/lib/python3.7/distutils/unixccompiler.py", line 199, in link
    raise LinkError(msg)
distutils.errors.LinkError: Command "gcc /tmp/tmpyq7p80or/tmp/tmpyq7p80or/source.o -L/home/vklein/odk/sage/local/li
b -llibraries -o /tmp/tmpyq7p80or/a.out" failed with exit status 1

Advices or help are welcome. I get the same errors with a sage with python2.7.

comment:28 in reply to: ↑ 26 Changed 16 months ago by jdemeyer

Replying to vklein:

numpy's issue #10500.

The solution seems to be recythonizing. Maybe we should just do that when building numpy.

comment:29 follow-up: Changed 16 months ago by vklein

Ok i will try.

comment:30 in reply to: ↑ 29 Changed 16 months ago by jdemeyer

Replying to vklein:

Ok i will try.

If you look in the setup.py file of numpy, you'll see

    from setuptools import setup
    if run_build:
        from numpy.distutils.core import setup
        cwd = os.path.abspath(os.path.dirname(__file__))
        if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
            # Generate Cython sources, unless building from source release
            generate_cython()

So I guess that removing the PKG-INFO file in spkg-install will force recythonization.

comment:31 Changed 16 months ago by vklein

Your trick almost work. The problem is that the numpy file tools/cythonize.py is not in numpy's source releases.

Last edited 16 months ago by vklein (previous) (diff)

comment:32 Changed 16 months ago by vklein

I confirm that numpy's install work if we remove PKG-INFO and add cythonize.py file.

comment:33 Changed 16 months ago by vklein

  • Branch changed from u/jdemeyer/upgrade_to_python_3_7_0 to u/vklein/upgrade_to_python_3_7_0

comment:34 Changed 16 months ago by vklein

  • Commit changed from 1bfcdc84ccc6bc18d36b2dd15a3daa0ae2e52ad2 to 6e0c5b41cf955cd1adfec7e10e5627682d652b33

Adding a patch to fix numpy's ​issue #10500.


New commits:

6e0c5b4Trac #25680: Patch numpy to fix issue with python3.7

comment:35 Changed 16 months ago by jdemeyer

Can we please deal with numpy in a separate ticket.

comment:36 follow-up: Changed 16 months ago by vklein

I don't know. The patch doesn't make sense without python 3.7.

comment:37 in reply to: ↑ 36 Changed 16 months ago by jdemeyer

Replying to vklein:

I don't know. The patch doesn't make sense without python 3.7.

It's needed for Python 3.7, but the fix is unrelated to it.

comment:38 Changed 16 months ago by jdemeyer

  • Status changed from needs_info to needs_work

If you're patching the sources anyway, I would prefer to patch setup.py instead of removing PKG-INFO.

comment:39 Changed 16 months ago by vklein

Sure ! It will be cleaner to patch setup.py.

comment:40 Changed 16 months ago by git

  • Commit changed from 6e0c5b41cf955cd1adfec7e10e5627682d652b33 to 1bfcdc84ccc6bc18d36b2dd15a3daa0ae2e52ad2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

comment:41 Changed 16 months ago by vklein

  • Dependencies changed from #25700 to #25700, #25755

comment:42 Changed 16 months ago by vklein

scipy errors seems pretty similar to numpy's:

scipy/cluster/_vq.c: In function '__Pyx__ExceptionSave':
    scipy/cluster/_vq.c:9728:19: error: 'PyThreadState {aka struct _ts}' has no member named 'exc_type'
         *type = tstate->exc_type;
                       ^
    scipy/cluster/_vq.c:9729:20: error: 'PyThreadState {aka struct _ts}' has no member named 'exc_value'
         *value = tstate->exc_value;

With likely the same cause. In setup.py file of scipy :

        if not os.path.exists(os.path.join(cwd, 'PKG-INFO')):
            # Generate Cython sources, unless building from source release
            generate_cython()
Last edited 16 months ago by vklein (previous) (diff)

comment:43 Changed 16 months ago by vklein

  • Dependencies changed from #25700, #25755 to #25700, #25755, #25759

comment:44 Changed 16 months ago by vklein

Building sage with this ticket and it's dependencies works fine.

I have made two run of sage -t -a --long, one with a sage python3.6 and one with this ticket. Comparing the logs of the two runs (and weakly idenfying a failure by the tuple (<sage file with the failure>, <line of the failure>) with a little script, i get the following results:

nb doctests failures with python3.6: 42990
nb doctests failures with python3.7: 43643
new python3.7 failures : 1366
python3.6 failures not appearing with python 3.7 : 713

Two exemple failures among 1366 :

File "src/sage/combinat/finite_state_machine.py", line 10190, in sage.combinat.finite_state_machine.FiniteStateMachine.asymptotic_moments
Failed example:
    moments = T.asymptotic_moments()
Exception raised:
    Traceback (most recent call last):
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 573, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 983, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.finite_state_machine.FiniteStateMachine.asymptotic_moments[2]>", line 1, in <module>
        moments = T.asymptotic_moments()
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/combinat/finite_state_machine.py", line 10540, in asymptotic_moments
        if not final_component.digraph().is_aperiodic():
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/graphs/digraph.py", line 3330, in is_aperiodic
        import networkx
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/__init__.py", line 114, in <module>
        import networkx.generators
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/generators/__init__.py", line 13, in <module>
        from networkx.generators.geometric import *
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/networkx/generators/geometric.py", line 27, in <module>
        from scipy.spatial import cKDTree as KDTree
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/scipy/spatial/__init__.py", line 95, in <module>
        from .qhull import *
      File "qhull.pyx", line 2224, in init scipy.spatial.qhull
    AttributeError: 'cython_function_or_method' object has no attribute '__func__'

sage -t --long src/sage/combinat/dyck_word.py
**********************************************************************
File "src/sage/combinat/dyck_word.py", line 873, in sage.combinat.dyck_word.DyckWord.plot
Failed example:
    w.plot()
Expected:
    Graphics object consisting of 1 graphics primitive
Got:
    doctest:warning
      File "/home/vklein/odk/sage/src/bin/sage-runtests", line 127, in <module>
        err = DC.run()
      File "/home/vklein/odk/sage/local/lib/python3.7/site-packages/sage/doctest/control.py", line 1172, in run
        self.run_doctests()
      ...
      File "/home/vklein/odk/sage/local/lib/python3.7/collections/__init__.py", line 52, in __getattr__
        DeprecationWarning, stacklevel=2)
            File "/home/vklein/odk/sage/local/lib/python3.7/warnings.py", line 99, in _showwarnmsg
        msg.file, msg.line)
    :
    DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
    Graphics object consisting of 1 graphics primitive

Note : To see the script and logs used https://github.com/vinklein/Trac25680

comment:45 Changed 15 months ago by jdemeyer

  • Dependencies changed from #25700, #25755, #25759 to #25700, #25755, #24766

comment:46 Changed 15 months ago by jdemeyer

  • Branch changed from u/vklein/upgrade_to_python_3_7_0 to u/jdemeyer/upgrade_to_python_3_7_0

comment:47 Changed 15 months ago by jdemeyer

  • Commit changed from 1bfcdc84ccc6bc18d36b2dd15a3daa0ae2e52ad2 to 24349a46b48fcd86f256200a954793b8b076260c

Python 3.7 seems to require libffi as new dependency, so I guess we should add that as Sage package.


New commits:

1c35bacUpgrade pexpect
7a7cb17Trac #25755: Patch numpy to fix issue with ptyhon3.7
6e291b1Trac #25755: update package version
48ceb84Trac #25755: recythonized numpy in every case
9c0cb67Add cython as dependency of numpy
7386ef8Update scipy to 1.1.0
b94a1bcRemove deprecated iprint parameter
0dcadccRemove scipy testsuite for now
e074fa3Merge commit '1c35bacd91a87db59d19ab78c2ecbfa17e38e240'; commit '9c0cb67a0d4a6f2b590143537c60d6e1d7cb84e8'; commit '0dcadcc4311e780e134c6912c3636c35e2d9d8b3' into t/25680/upgrade_to_python_3_7_0
24349a4Upgrade to Python 3.7.0

comment:48 Changed 15 months ago by jdemeyer

  • Dependencies changed from #25700, #25755, #24766 to #25700, #25755, #24766, #25900

comment:49 Changed 15 months ago by git

  • Commit changed from 24349a46b48fcd86f256200a954793b8b076260c to 2f7648b2a74d32a4c815f0e2e9272df59bc0cdfe

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8f6d830Add libffi package
3b8e438Merge commit '8f6d8304420e0894d5e8221238adcb09ab7edbed' into t/25680/upgrade_to_python_3_7_0
2f7648bUpgrade to Python 3.7.0

comment:50 Changed 15 months ago by git

  • Commit changed from 2f7648b2a74d32a4c815f0e2e9272df59bc0cdfe to 4e1b2bef2a974b1d58e6469e225a54be6d88c36b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4e1b2beUpgrade to Python 3.7.0

comment:51 in reply to: ↑ 15 Changed 15 months ago by jdemeyer

Replying to fbissey:

Failure to build _ctypes usually means a problem with ffi (sometimes called libffi) may be there is minimum version requirement?

It seems that older Python versions vendored libffi but this was changed in Python 3.7. See https://docs.python.org/3/whatsnew/3.7.html#build-changes

comment:52 Changed 15 months ago by jdemeyer

  • Status changed from needs_work to needs_review

comment:53 Changed 15 months ago by vklein

The build works fine with the new dependencies.

comment:54 Changed 15 months ago by embray

  • Milestone changed from sage-8.3 to sage-8.4

Ok, since this will be delayed until 8.4 (at least) I rescind my earlier concerns that this will hinder the Python 3 port. After a thorough reading of the release notes I don't think there's anything that will hurt, and there are some things that will almost certainly help (such a dict insertion order preservation).

I just need to see how Python 3.7 does on Cygwin (originally I was aiming to have CPython fully supported on Cygwin by 3.7, but I just didn't have the time, though we did get some important big fixes in).

comment:55 Changed 14 months ago by chapoton

  • Status changed from needs_review to needs_work

branch does not apply

comment:56 Changed 13 months ago by vklein

@embray Have you some news on Python3.7 on Cygwin ?

comment:57 Changed 13 months ago by embray

My only news is I have multiple upstream PRs necessary for 3.7 to at least build, at a bare minimum, on Cygwin, and they have been stalled for quite a bit. I was just thinking yesterday of poking python-dev soon for someone to look at them.

I actually now maintain an official CPython buildbot for Cygwin, but it's all but useless until those fixes are accepted: https://buildbot.python.org/all/#/builders/164

comment:58 Changed 13 months ago by embray

The more I think about it, the more I'm also hesitant to make 3.7 the Python for Sage until 3.7 is in more distros. But I'm not sure. It would be easier to depend on an arbitrary Python 3 (or at least 3.4+) if we were also dropping Python 2 support completely. I don't think we've set a schedule for doing that, but I doubt that will happen in Sage until after the Python 2 EoL.

comment:59 follow-up: Changed 13 months ago by vklein

Ok i see (https://github.com/python/cpython/pulls/embray).

So does that mean you think we should fix problems like itertools.islice in sage rather than waiting for 3.7 ?

comment:60 in reply to: ↑ 59 Changed 13 months ago by embray

Replying to vklein:

Ok i see (https://github.com/python/cpython/pulls/embray).

Wow, I have several PRs pending... FWIW the only two that are major blockers for Python on Cygwin are:

https://github.com/python/cpython/pull/4348 https://github.com/python/cpython/pull/8712

Of course, that alone doesn't necessarily need to block Sage's Python 3 spkg--as it is I believe we already include some custom patches for Cygwin, and I'm okay with that for now. I just brought it up since you asked about the status on Cygwin. Also, beyond those patches, there may be other issues. I actually haven't spent much time on it since I was hoping to get some movement on those bare minimum issues before I do much more... *sigh*

So does that mean you think we should fix problems like itertools.islice in sage rather than waiting for 3.7 ?

Yes, and for two reasons:

1) We've already fixed those problems in several cases; probably even the majority of them (I have a few more in my Python 3 branch I haven't even made tickets for yet).

2) We will likely have to work around that anyway for downstream distros, unless we want to make 3.7 a hard requirement for Sage. But that might be tough, at least in the near term...

comment:61 Changed 12 months ago by slelievre

  • Description modified (diff)
  • Keywords upgrade Python3 added
  • Milestone changed from sage-8.4 to sage-wishlist
  • Summary changed from Upgrade to Python 3.7.0 to Upgrade to Python 3.7.x

Python 3.7.1 was released.

comment:62 Changed 10 months ago by slelievre

Python 3.7.2rc1 was released: Python insider blog post: Python 3.7.2rc1.

comment:63 Changed 8 months ago by jdemeyer

  • Description modified (diff)
  • Milestone changed from sage-wishlist to sage-8.7

comment:64 Changed 8 months ago by git

  • Commit changed from 4e1b2bef2a974b1d58e6469e225a54be6d88c36b to 3d986766849a48dc87ae51acd9d7803f0896736a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3d98676Upgrade to Python 3.7.2

comment:65 Changed 8 months ago by jdemeyer

  • Dependencies #25700, #25755, #24766, #25900 deleted
  • Status changed from needs_work to needs_review

comment:66 follow-up: Changed 8 months ago by vklein

When running sage in py3 a difference with python3.6 is that we get a bunch of doctests errors of type DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working. They mainly comes form matplotlib package but also from packages jinja2, networkx and werkzeug.

comment:67 in reply to: ↑ 66 Changed 8 months ago by jdemeyer

Replying to vklein:

When running sage in py3 a difference with python3.6 is that we get a bunch of doctests errors of type DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working. They mainly comes form matplotlib package but also from packages jinja2, networkx and werkzeug.

We could just hide that warning.

comment:68 Changed 8 months ago by vklein

Python version Files having failures failing doctests
3.6.6 166 717
3.7.2 387 1104

Numbers of DeprecationWarning: Using or importing the ABCs from 'collections'... : 230.

new python3.7 failures : 407

python3.6 failures not appearing with python 3.7 : 18

Link for the doctest logs here

Last edited 8 months ago by vklein (previous) (diff)

comment:69 Changed 8 months ago by jdemeyer

By the way, I don't think that increasing the number of doctest failures should prevent this ticket from being merged. We'll need to support Python 3.7 sooner or later, so we might as well do it sooner.

comment:70 Changed 8 months ago by vklein

Yes i agree with that. I am ok to give positive review for this ticket as you can now run sage with this python version.

@embray can you confirm that it's ok for Cygwin now ? ((https://github.com/python/cpython/pull/4348https://github.com/python/cpython/pull/8712 have been merged)

comment:71 follow-up: Changed 8 months ago by chapoton

oh, great, 400 new doctests to fix.. really great.. I was afraid to start lacking things to do...

Does this pass the python3 test target ?

If not, it will fail on the buildbot that checks exactly that.. and maybe the release manager will say niet.

comment:72 Changed 8 months ago by jdemeyer

Like I said, we would need to fix those failures at some point anyway. I don't think that we want to stay on Python 3.6 forever.

comment:73 Changed 8 months ago by git

  • Commit changed from 3d986766849a48dc87ae51acd9d7803f0896736a to 99cbc2d0705662afa3e7a971b5b9f1fd12aa85f3

Branch pushed to git repo; I updated commit sha1. New commits:

99cbc2dDisable collections.abc warnings

comment:74 Changed 8 months ago by jdemeyer

Most remaining failures are of the form

sage -t --long src/sage/rings/number_field/galois_group.py
**********************************************************************
File "src/sage/rings/number_field/galois_group.py", line 20, in sage.rings.number_field.galois_group
Failed example:
    G == loads(dumps(G))
Exception raised:
    Traceback (most recent call last):
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.number_field.galois_group[3]>", line 1, in <module>
        G == loads(dumps(G))
      File "sage/misc/persist.pyx", line 968, in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7256)
        return unpickler.load()
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py", line 573, in extend
        self._require_mutable()
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py", line 740, in _require_mutable
        if self._is_immutable:
      File "/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py", line 879, in __getattr__
        raise AttributeError("'Sequence_generic' object has no attribute '%s'"%name)
    AttributeError: 'Sequence_generic' object has no attribute '_is_immutable'
**********************************************************************

comment:75 Changed 8 months ago by vklein

The filter works ! So it will be more something like 177 new doctests failures rather than 407.

comment:76 Changed 8 months ago by jdemeyer

Also

DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead

comment:77 Changed 8 months ago by jdemeyer

  • Dependencies set to #27269

comment:78 in reply to: ↑ 71 Changed 8 months ago by jdemeyer

Replying to chapoton:

oh, great, 400 new doctests to fix.. really great.. I was afraid to start lacking things to do...

It's not as bad as it sounds. Those 400 doctests are mostly just instances of either of two failures. So fixing those two should bring that number down to a very small number.

comment:79 follow-up: Changed 8 months ago by chapoton

does not build on my patchbot (and others)

comment:80 in reply to: ↑ 79 Changed 8 months ago by jdemeyer

Replying to chapoton:

does not build

Details please...

comment:81 Changed 8 months ago by fbissey

The only thing I can see from the patch bot is that the python 3.7 tarball is not mirrored so the bots fail to fetch it.

comment:82 Changed 8 months ago by chapoton

yes, sorry for the noise..

Is there good hope to fix here the

AttributeError: 'Sequence_generic' object has no attribute '_is_immutable

errors ? I remember that the Galois group code is a bit antique and somewhat ugly..

comment:83 Changed 8 months ago by jdemeyer

  • Dependencies #27269 deleted

comment:84 Changed 8 months ago by git

  • Commit changed from 99cbc2d0705662afa3e7a971b5b9f1fd12aa85f3 to f7bb9577fe1d233c3cb11e5f2a993a61fef16e3f

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d4a4913Upgrade to Python 3.7.2
f7bb957Disable collections.abc warnings

comment:85 Changed 8 months ago by jdemeyer

I'm looking into this:

sage: L = Sequence(range(10))
sage: L
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
sage: loads(dumps(L))
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-21e74d398e6e> in <module>()
----> 1 loads(dumps(L))

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/misc/persist.pyx in sage.misc.persist.loads (build/cythonized/sage/misc/persist.c:7256)()
    966 
    967     unpickler = SageUnpickler(io.BytesIO(s))
--> 968     return unpickler.load()
    969 
    970 

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py in extend(self, iterable)
    571             [1, 2, 3, 0, 1, 2, 3]
    572         """
--> 573         self._require_mutable()
    574         v = [self.__universe(x) for x in iterable]
    575         list.extend(self, v)

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py in _require_mutable(self)
    738             ValueError: object is immutable; please change a copy instead.
    739         """
--> 740         if self._is_immutable:
    741             raise ValueError("object is immutable; please change a copy instead.")
    742 

/home/dimpase/sage-python3/local/lib/python3.7/site-packages/sage/structure/sequence.py in __getattr__(self, name)
    877             return self.__hash
    878         else:
--> 879             raise AttributeError("'Sequence_generic' object has no attribute '%s'"%name)
    880 seq = Sequence
    881 

AttributeError: 'Sequence_generic' object has no attribute '_is_immutable'

This is causing the cast majority of all new doctest failures.

comment:86 Changed 8 months ago by git

  • Commit changed from f7bb9577fe1d233c3cb11e5f2a993a61fef16e3f to 830749c304e408cf8b4825f98649a990519495fd

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e233a9eImplement pickling for Sequence_generic
0d4139cUpgrade to Python 3.7.2
830749cDisable collections.abc warnings

comment:87 Changed 8 months ago by jdemeyer

  • Dependencies set to #27334

comment:88 Changed 8 months ago by jdemeyer

There is just one failure with ptest-python3: a timeout in src/sage/parallel/map_reduce.py

comment:89 Changed 8 months ago by vklein

Is this timeout somehow related with python3.7 ? If not with #27334 i am ok to set this ticket in positive review.

comment:90 Changed 8 months ago by jdemeyer

I'm not entirely sure what is causing that timeout... but I also don't think that it should prevent merging this ticket.

comment:91 Changed 8 months ago by vklein

  • Status changed from needs_review to positive_review

Ok then.

comment:92 Changed 8 months ago by vklein

  • Reviewers set to Vincent Klein, Eric Bray

comment:93 Changed 8 months ago by jhpalmieri

  • Reviewers changed from Vincent Klein, Eric Bray to Vincent Klein, Erik Bray

comment:94 Changed 8 months ago by chapoton

Does this pass the refreshed python3 test target of 8.7.b5 (that checks more modules)?

If not, it will fail on the buildbot that checks exactly that.. and maybe the release manager will say niet.

comment:95 Changed 8 months ago by jdemeyer

I would argue that we can always change the list of passing doctests in this ticket. Otherwise, we are aiming for a moving target and this ticket won't ever be fixed.

comment:96 Changed 8 months ago by chapoton

yes, sure. I would very much like this ticket to be fixed, and I do not want to do anything against it. But still maybe somebody should check if the passing doctests needs to be changed ?

Last edited 8 months ago by chapoton (previous) (diff)

comment:97 Changed 8 months ago by vbraun

  • Branch changed from u/jdemeyer/upgrade_to_python_3_7_0 to 830749c304e408cf8b4825f98649a990519495fd
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:98 Changed 8 months ago by vbraun

  • Branch changed from 830749c304e408cf8b4825f98649a990519495fd to u/jdemeyer/upgrade_to_python_3_7_0
  • Resolution fixed deleted
  • Status changed from closed to new

Ironically, this fails on py3:

sage -t --long src/sage_setup/docbuild/__init__.py
**********************************************************************
File "src/sage_setup/docbuild/__init__.py", line 98, in sage_setup.docbuild.builder_helper
Failed example:
    build_many(build_ref_doc, [("docname", "en", "html", {})])
Expected:
    Traceback (most recent call last):
    ...
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation',))
Got:
    <BLANKLINE>
    multiprocessing.pool.RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 129, in f
        runsphinx()
      File "<doctest sage_setup.docbuild.builder_helper[3]>", line 2, in raiseBaseException
        raise BaseException("abort pool operation")
    BaseException: abort pool operation
    <BLANKLINE>
    During handling of the above exception, another exception occurred:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
        result = (True, func(*args, **kwds))
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
        return list(map(*args))
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 73, in build_ref_doc
        getattr(ReferenceSubBuilder(doc, lang), format)(*args, **kwds)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 742, in _wrapper
        getattr(DocBuilder, build_type)(self, *args, **kwds)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 138, in f
        raise Exception("Non-exception during docbuild: %s"%(e,), e)
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))
    """
    <BLANKLINE>
    The above exception was the direct cause of the following exception:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.docbuild.builder_helper[7]>", line 1, in <module>
        build_many(build_ref_doc, [("docname", "en", "html", {})])
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 282, in build_many
        ret = x.get(99999)
      File "/var/lib/buildbot/slave/sage3_git/build/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
        raise self._value
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))
**********************************************************************
1 item had failures:
   1 of  10 in sage_setup.docbuild.builder_helper
    [36 tests, 1 failure, 0.65 s]

comment:99 Changed 8 months ago by jhpalmieri

Here's what I got for the test suite on OS X:

Ran 93 tests in 225.537s

FAILED (errors=3, skipped=1)
8 tests failed again:
    test_normalization test_robotparser test_site test_ssl
    test_sysconfig test_urllib test_urllib2 test_urllib2_localnet

== Tests result: FAILURE then FAILURE ==

389 tests OK.

8 tests failed:
    test_normalization test_robotparser test_site test_ssl
    test_sysconfig test_urllib test_urllib2 test_urllib2_localnet

16 tests skipped:
    test_dbm_gnu test_devpoll test_epoll test_gdb test_msilib
    test_multiprocessing_fork test_ossaudiodev test_spwd
    test_startfile test_tix test_tk test_ttk_guionly test_winconsoleio
    test_winreg test_winsound test_zipfile64

8 re-run tests:
    test_normalization test_robotparser test_site test_ssl
    test_sysconfig test_urllib test_urllib2 test_urllib2_localnet

3 tests run no tests:
    test_dtrace test_future4 test_largefile

Total duration: 48 min 44 sec
Tests result: FAILURE then FAILURE
make[2]: *** [test] Error 2
An error occurred while testing Python

comment:100 follow-up: Changed 8 months ago by embray

  • Reviewers changed from Vincent Klein, Erik Bray to Vincent Klein
  • Status changed from new to needs_info

Please also don't set this back to positive review until I've had a chance to try it. I didn't even know this had been closed. I don't know why you added me to the reviewers when I haven't even commented on this ticket in 5 months (note: I'm not opposed to it but I haven't even looked at it since it started having activity again).

comment:101 Changed 8 months ago by embray

  • Status changed from needs_info to needs_review

(For some reason the workflow was in an odd state where I couldn't set it directly back to needs review)

comment:102 Changed 8 months ago by embray

There are two critical fixes needed for Python 3.7 to even build on Cygwin:

  • One of them has been backported to the 3.7 branch, and I confirmed is in 3.7.2.
  • The other has not been backported to 3.7, which is annoying. I'm not sure why not; I'll have to look into what I have to do to request a backport. In the meantime this patch needs to be included, at the very least.

comment:103 in reply to: ↑ 100 Changed 8 months ago by vklein

Replying to embray:

Please also don't set this back to positive review until I've had a chance to try it. I didn't even know this had been closed. I don't know why you added me to the reviewers when I haven't even commented on this ticket in 5 months (note: I'm not opposed to it but I haven't even looked at it since it started having activity again).

Sorry but i thought you already had validated 3.7.2 for OSX. I was wrong.

comment:104 Changed 8 months ago by jhpalmieri

For comparison, here is what happens when I run the tests with Python 3.6.6:

Ran 93 tests in 225.364s

FAILED (errors=3, skipped=1)
10 tests failed again:
    test_asyncio test_normalization test_pyexpat test_robotparser
    test_site test_ssl test_sysconfig test_urllib test_urllib2
    test_urllib2_localnet

== Tests result: FAILURE then FAILURE ==

382 tests OK.

10 tests failed:
    test_asyncio test_normalization test_pyexpat test_robotparser
    test_site test_ssl test_sysconfig test_urllib test_urllib2
    test_urllib2_localnet

15 tests skipped:
    test_dbm_gnu test_devpoll test_epoll test_gdb test_msilib
    test_ossaudiodev test_spwd test_startfile test_tix test_tk
    test_ttk_guionly test_winconsoleio test_winreg test_winsound
    test_zipfile64

10 re-run tests:
    test_asyncio test_normalization test_pyexpat test_robotparser
    test_site test_ssl test_sysconfig test_urllib test_urllib2
    test_urllib2_localnet

Total duration: 49 min 37 sec
Tests result: FAILURE then FAILURE
make[2]: *** [test] Error 2
An error occurred while testing Python

comment:105 follow-up: Changed 8 months ago by chapoton

The second Cygwin fix seems to have been merged, no ?

https://github.com/python/cpython/pull/12063

comment:106 in reply to: ↑ 105 Changed 8 months ago by embray

Replying to chapoton:

The second Cygwin fix seems to have been merged, no ?

https://github.com/python/cpython/pull/12063

Yes, 6 days ago, after I made noise about it.

comment:107 Changed 8 months ago by chapoton

Should we wait for a release of python 3.7 ?

comment:108 Changed 8 months ago by embray

Not necessarily, we can still include the patch now.

comment:109 Changed 8 months ago by chapoton

  • Dependencies #27334 deleted

comment:110 Changed 7 months ago by git

  • Commit changed from 830749c304e408cf8b4825f98649a990519495fd to b94ad2c2e95900b66619deeeb1ce43d234adbe28

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8c12c27Upgrade to Python 3.7.2
bb0564dDisable collections.abc warnings
b94ad2cFix "abort pool operation" docbuild doctest

comment:111 Changed 7 months ago by git

  • Commit changed from b94ad2c2e95900b66619deeeb1ce43d234adbe28 to a42152cab4f24c6db48a0263ccef9f68da151325

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b08639cUpgrade to Python 3.7.2
2d6bb9aDisable collections.abc warnings
a42152cFix "abort pool operation" docbuild doctest

comment:112 Changed 7 months ago by git

  • Commit changed from a42152cab4f24c6db48a0263ccef9f68da151325 to e5f3023841c01b369458244ccd7821b8911323e3

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

36c93f1Upgrade to Python 3.7.2
b03a9f8Disable collections.abc warnings
e5f3023Fix "abort pool operation" docbuild doctest

comment:113 Changed 7 months ago by jdemeyer

I'm getting a lot of failures due to

sage: gap_reset_workspace()
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-01843f720c90> in <module>()
----> 1 gap_reset_workspace()

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in gap_reset_workspace(max_workspace_size, verbose)
   1577     # Create new workspace with filename WORKSPACE
   1578     g = Gap(use_workspace_cache=False, max_workspace_size=None)
-> 1579     g.eval('SetUserPreference("HistoryMaxLines", 30)')
   1580     from sage.tests.gap_packages import all_installed_packages
   1581     for pkg in all_installed_packages():

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in eval(self, x, newlines, strip, split_lines, **kwds)
    581             if not input_line.endswith(';'):
    582                 input_line += ';'
--> 583         result = Expect.eval(self, input_line, **kwds)
    584         if not newlines:
    585             result = result.replace("\\\n","")

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in eval(self, code, strip, synchronize, locals, allow_use_file, split_lines, **kwds)                                                                                                                                                                    
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in <listcomp>(.0)
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in _eval_line(self, line, allow_use_file, wait_for_prompt, restart_if_needed)
    741         try:
    742             if self._expect is None:
--> 743                 self._start()
    744             E = self._expect
    745             #import pdb; pdb.set_trace()

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in _start(self)
   1272 
   1273         # set random seed
-> 1274         self.set_seed(self._seed)
   1275 
   1276     def _function_class(self):

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in set_seed(self, seed)
   1177         if seed is None:
   1178             seed = self.rand_seed()
-> 1179         self.eval("Reset(GlobalMersenneTwister,%d)" % seed)
   1180         self.eval("Reset(GlobalRandomSource,%d)" % seed)
   1181         self._seed = seed

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in eval(self, x, newlines, strip, split_lines, **kwds)
    581             if not input_line.endswith(';'):
    582                 input_line += ';'
--> 583         result = Expect.eval(self, input_line, **kwds)
    584         if not newlines:
    585             result = result.replace("\\\n","")

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in eval(self, code, strip, synchronize, locals, allow_use_file, split_lines, **kwds)
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/expect.py in <listcomp>(.0)
   1349                 elif split_lines:
   1350                     return '\n'.join([self._eval_line(L, allow_use_file=allow_use_file, **kwds)
-> 1351                                         for L in code.split('\n') if L != ''])
   1352                 else:
   1353                     return self._eval_line(code, allow_use_file=allow_use_file, **kwds)

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/interfaces/gap.py in _eval_line(self, line, allow_use_file, wait_for_prompt, restart_if_needed)
    753             # garbage should be filtered out by this point); here we decode
    754             # them (on Python 3), currently just using the default encoding
--> 755             normal, error = bytes_to_str(normal), bytes_to_str(error)
    756 
    757             if len(error):

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/cpython/string.c:1515)()
     29 
     30 
---> 31 cpdef inline str bytes_to_str(b, encoding=None, errors=None):
     32     r"""
     33     Convert ``bytes`` to ``str``.

/home/jdemeyer/sage-python3/local/lib/python3.7/site-packages/sage/cpython/string.pxd in sage.cpython.string.bytes_to_str (build/cythonized/sage/cpython/string.c:1399)()
     58         return <str>b
     59     else:
---> 60         return _cstr_to_str(<bytes>b, encoding, errors)
     61 
     62 

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 15: invalid start byte

comment:114 Changed 7 months ago by jdemeyer

But why is GAP producing a byte 0x80 in its output?

comment:115 follow-up: Changed 7 months ago by chapoton

what is the locale used ?

comment:116 in reply to: ↑ 115 Changed 7 months ago by jdemeyer

Replying to chapoton:

what is the locale used ?

I have no idea, how do I figure that out?

comment:117 Changed 7 months ago by jdemeyer

In any case, Python 3 was working quite well on this system before. So it's a regression since a few betas ago.

comment:118 Changed 7 months ago by chapoton

Maybe using

echo $LANG
echo $LC_ALL

comment:119 follow-up: Changed 7 months ago by chapoton

Does the current branch include the patches required for Cygwin ?

comment:120 in reply to: ↑ 119 Changed 7 months ago by jdemeyer

Replying to chapoton:

Does the current branch include the patches required for Cygwin ?

Yes, I added the patches that @embray mentioned.

I have not looked further into the GAP issue. Maybe somebody else should try it.

comment:121 Changed 7 months ago by jhpalmieri

I don't see the GAP problem when I do a Python 3 build with this branch on OS X:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.7.beta7, Release Date: 2019-03-10               │
│ Using Python 3.7.2. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: gap_reset_workspace()
sage: 

and

$ ./sage -t src/sage/interfaces/gap.py 
too many failed tests, not using stored timings
Running doctests with ID 2019-03-13-13-25-14-a8d0606b.
Git branch: t/25680/upgrade_to_python_3_7_0
Using --optional=dochtml,mpir,python2,sage
Doctesting 1 file.
sage -t src/sage/interfaces/gap.py
    [222 tests, 6.84 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 8.3 seconds
    cpu time: 1.9 seconds
    cumulative wall time: 6.8 seconds
$ ./sage -t src/sage/interfaces/gap_workspace.py
too many failed tests, not using stored timings
Running doctests with ID 2019-03-13-13-25-35-b861aafd.
Git branch: t/25680/upgrade_to_python_3_7_0
Using --optional=dochtml,mpir,python2,sage
Doctesting 1 file.
sage -t src/sage/interfaces/gap_workspace.py
    [8 tests, 0.01 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 0.0 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

comment:122 follow-up: Changed 7 months ago by chapoton

I have two questions here:

  • do we have the approval of Erik (about Windows) ?
  • does the branch pass all the tests in the current python3 target ?

EDIT: I have no problem with gap either.

Last edited 7 months ago by chapoton (previous) (diff)

comment:123 Changed 7 months ago by jdemeyer

I reduced the problem to the shell command

echo 'Reset(GlobalMersenneTwister,500837550);' | ./sage --gap -r -b -p -T -E local/share/sage/ext/gap/sage.g

This is outputting binary garbage which Python 3 cannot convert to a string.

I don't think it's caused by this ticket, but it does prevent me from working on Python 3.

comment:124 Changed 7 months ago by jdemeyer

Even in an interactive GAP session, I'm getting the same binary garbage:

 *********   GAP 4.10.0 of 01-Nov-2018
 *  GAP  *   https://www.gap-system.org
 *********   Architecture: x86_64-pc-linux-gnu-default64
 Configuration:  gmp 6.0.0, readline
 Loading the library and packages ...
 Packages:   Alnuth 3.1.0, AtlasRep 1.5.1, AutPGrp 1.10, CRISP 1.4.4, CTblLib 1.2.2, FactInt 1.6.2, FGA 1.4.0, GAPDoc 1.6.2, IRREDSOL 1.4, LAGUNA 3.9.0, 
             Polycyclic 2.14, PrimGrp 3.3.2, SmallGrp 1.3, Sophus 1.24, TomLib 1.2.7, TransGrp 2.0.4
 Try '??help' for help. See also '?copyright', '?cite' and '?authors'
gap> Reset(GlobalMersenneTwister,0);
�>[>�1P[�\"�2�ʦ\005�t>���T��\020@�8��l���\023�%��\023hK�q\033�'�\026fB\025��P��\n~otM��^�����É)\<[\000\b\<G\rs���\n���Q[LN�\030\<�\013�L~8\
x*\t�x�p������l\b�\022P`���^��\021p�=j��I8j��rV\023�~N�^l�9�'�Ᾱ�@^LI[\
I\027��j\023���#\03122q\034���Uv��n�p��\\014�H\bׯ2�9W�.�Ni\034���4J�Tb_�\�2\013�I\000JM�n6ODt`_\007!\023:�*�������_*�qj4�kʪ�-F<pr��H��xR\005�uS�\026\n\
�\027���'\005FT��a&]\024�\026���d���#\rK��)k�\\��L��I��\033o�F��BY D\>�\006tF���^\032-\n��z���f\020>��p�\004K<�
                                                                                                               �lCJ�o\0278|\027P3�Q%D�Q��V\bU
�hR-���]�~N���\c�a�\025[O\036�\037MJt��5�i�\014\bu=)=f��\034\c��\>����*��\006��{o\
\016jws�1�&we5�[�BFǴzl�'�_Q�\021[x�R[�\021\t����7F˵0[N\t���U�/��UP\005\t������\024���IנL(��7��9��l`��L�\027�Pè�8�TdO,r��`RY\\W\026s�h]\c�j|�d���\016�8�@\
�\027�\006�2.�\005�\n6���7`\005[-)\037�N4�=�W��N�+�\025�2a��\037Tʹ�\0164:*k��z�\c,vԹ��&,\025\<&>Pk
�4��\005����I_��Q�Z����f(2�\Iw��\023H�p�\033\c\\\016�����PTd�C\020��タ�8b\025.����\034qc��\037?���)�6'Kd�\033$��Mz���!t;Njn�whȔ5�#D_�� �!<ؾϛ\031O���\
��2�,h8=\blJ$����ͫ��)���\031-<4��z���dj\<�\t<;�g�&u�<GS\027�2E�|[�0{\t�K�\022}
                                                                             \030��\026cW�e�\020���B��C�?]k&��`�\026�!\017\>\ru5�j) �ܮ�Y\005��� \004\
���C�7        �\>n�oҤ\\\000�\>,�\026dJ�n)S#!l�/�8?�7Q\004Jt�ɭ7�\>*B`jmT�g��\036S|]\006\033�\>|\007\004�\037\033                  �EV�&��\bS�P�R\
/߾��ɢ��a\031{i�\r\020N\030��\032)�&��\"{�־�.���~,��\034����I6\030\034f�\032����(4�^`m&w��\000�D\016,\b�[7/ �\025`\030s@ZW;�\024���æMү4�Hc�\033�\"��\006\<O-�FP흛-�A��KW2>p��������q\b�\022�J*\031P��^���yJ�����\016H�C9�,|�j�?�[U��\c\                            �*�j���W[\023|`��.���h�9e{sy�\rw��\b4�[z\023���m\022�\027ȯ[ЩOž\
\036�޾> J\b�G�4;�\r��W\022i���zs���zW=\004���yT�\007�����Lr30��\005�v�%D�\025�\032\037\c\b
                      1�Ft�1�-�\033[Z\035��t���\030YY�-V�o�g���q\007Lb�'J.R�4jC�C���U%\>vƸ%C]�<��[iK#8.�j>O1T��\cl��[�����B?_\0277�\004\0275�\000d���X\
\014\013�\032ijv�.u��\016+��\017�:_������esͺ\023�H\<z�\n\023E���\r����\024k�\n�W�T(����k8\032rJ|=8\"�|�\004gU�׭����L=;�ow�Ҿ`:,\>�mdQ-Ԫ�Bqy}�-��bA{p\<\
\000\0001234" ]

comment:125 Changed 7 months ago by jdemeyer

  • Dependencies set to #27523

comment:126 Changed 7 months ago by git

  • Commit changed from e5f3023841c01b369458244ccd7821b8911323e3 to 10442de1f848010ec373ea5afa9332885ffaf59c

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f74ecc0GAP: silence output from Reset(GlobalMersenneTwister)
93dfaafUpgrade to Python 3.7.2
700e513Disable collections.abc warnings
10442deFix "abort pool operation" docbuild doctest

comment:127 in reply to: ↑ 122 Changed 7 months ago by embray

Replying to chapoton:

I have two questions here:

  • do we have the approval of Erik (about Windows) ?
  • does the branch pass all the tests in the current python3 target ?

EDIT: I have no problem with gap either.

For myself, I've been waiting until some of the other issues with this branch are worked out--i.e. that it's otherwise good-to-go on Linux.

But if you think that's nearly the case I should start now, since building on Windows takes some time.

comment:128 follow-up: Changed 7 months ago by chapoton

I have got 3 doctests failure on the python3 test-suite:

sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
sage -t --long src/sage/parallel/map_reduce.py  # Timed out after testing finished

EDIT: in particular, the "abort pool operation" is not fixed as claimed.

Last edited 7 months ago by chapoton (previous) (diff)

comment:129 in reply to: ↑ 128 Changed 7 months ago by embray

Replying to chapoton:

I have got 3 doctests failure on the python3 test-suite:

sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
sage -t --long src/sage/parallel/map_reduce.py  # Timed out after testing finished

The first one I don't know about.

The second one I think Jeroen just fixed: https://git.sagemath.org/sage.git/commit/?id=10442de1f848010ec373ea5afa9332885ffaf59c

The third one is a little concerning: That module tends to have some random failures in it and I have made a few fixes to it in the past. Nevertheless apparently it still has bugs. Though I'm surprised it's timing out (albeit after testing finished, meaning it's probably hanging during interpreter shutdown...

comment:130 Changed 7 months ago by chapoton

on the very latest branch, I still get

sage -t --long src/sage_setup/docbuild/__init__.py
**********************************************************************
File "src/sage_setup/docbuild/__init__.py", line 105, in sage_setup.docbuild.builder_helper
Failed example:
    try:
        build_many(build_ref_doc, [("docname", "en", "html", {})])
    except Exception as E:
        "Non-exception during docbuild: abort pool operation" in E
Exception raised:
    multiprocessing.pool.RemoteTraceback: 
    """
    Traceback (most recent call last):
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 134, in f
        runsphinx()
      File "<doctest sage_setup.docbuild.builder_helper[3]>", line 2, in raiseBaseException
        raise BaseException("abort pool operation")
    BaseException: abort pool operation

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/chapoton/sage3/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
        result = (True, func(*args, **kwds))
      File "/home/chapoton/sage3/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
        return list(map(*args))
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 80, in build_ref_doc
        getattr(ReferenceSubBuilder(doc, lang), format)(*args, **kwds)
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 761, in _wrapper
        getattr(DocBuilder, build_type)(self, *args, **kwds)
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 143, in f
        raise Exception("Non-exception during docbuild: %s"%(e,), e)
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))
    """

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<doctest sage_setup.docbuild.builder_helper[7]>", line 2, in <module>
        build_many(build_ref_doc, [("docname", "en", "html", {})])
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage_setup/docbuild/__init__.py", line 289, in build_many
        ret = x.get(99999)
      File "/home/chapoton/sage3/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
        raise self._value
    Exception: ('Non-exception during docbuild: abort pool operation', BaseException('abort pool operation'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/chapoton/sage3/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage_setup.docbuild.builder_helper[7]>", line 4, in <module>
        "Non-exception during docbuild: abort pool operation" in E
    TypeError: argument of type 'Exception' is not iterable
**********************************************************************
1 item had failures:
   1 of   9 in sage_setup.docbuild.builder_helper
    [35 tests, 1 failure, 0.49 s]
----------------------------------------------------------------------
sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
----------------------------------------------------------------------

and

sage -t --long src/sage/interacts/debugger.py
**********************************************************************
File "src/sage/interacts/debugger.py", line 102, in sage.interacts.debugger.Debug.curframe
Failed example:
    d.curframe()
Expected:
    <frame object at 0x...>
Got:
    <repr(<sage.interfaces.sage0.SageElement at 0x7f0c32222fc0>) failed: TypeError: replace() argument 2 must be str, not SageFunction>
**********************************************************************
1 item had failures:
   1 of   4 in sage.interacts.debugger.Debug.curframe
    [23 tests, 1 failure, 1.36 s]
----------------------------------------------------------------------
sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
----------------------------------------------------------------------

EDIT: The failing doctest in debugger works fine when run interactively.

Last edited 7 months ago by chapoton (previous) (diff)

comment:131 Changed 7 months ago by chapoton

Here is fix for the docbuild doctest, please add to your branch if you agree:

-- a/src/sage_setup/docbuild/__init__.py
+++ b/src/sage_setup/docbuild/__init__.py
@@ -105,7 +105,7 @@ def builder_helper(type):
         sage: try:
         ....:     build_many(build_ref_doc, [("docname", "en", "html", {})])
         ....: except Exception as E:
-        ....:     "Non-exception during docbuild: abort pool operation" in E
+        ....:     "Non-exception during docbuild: abort pool operation" in str(E)
         True
     """
     def f(self, *args, **kwds):

comment:132 Changed 7 months ago by git

  • Commit changed from 10442de1f848010ec373ea5afa9332885ffaf59c to 7050a4b5d585c3b775244964678eeea546069e01

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

7050a4bFix "abort pool operation" docbuild doctest

comment:133 Changed 7 months ago by git

  • Commit changed from 7050a4b5d585c3b775244964678eeea546069e01 to eed6a11a07b5efdff43ef711046e07e95fe5a045

Branch pushed to git repo; I updated commit sha1. New commits:

eed6a11py3: fix for repr of frame objects

comment:134 Changed 7 months ago by jdemeyer

I'm now down to one failure: the timeout in map_reduce.py

comment:135 follow-up: Changed 7 months ago by chapoton

The debugger failure is still there for me.

comment:136 in reply to: ↑ 135 Changed 7 months ago by jdemeyer

Replying to chapoton:

The debugger failure is still there for me.

Can you post the complete output of ./sage -t --long src/sage/interacts/debugger.py?

I get

too few successful tests, not using stored timings
Running doctests with ID 2019-03-21-15-27-35-0a07308a.
Git branch: HEAD
Using --optional=dochtml,memlimit,mpir,python2,sage
Doctesting 1 file.
sage -t --long src/sage/interacts/debugger.py
    [23 tests, 2.38 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 2.4 seconds
    cpu time: 0.1 seconds
    cumulative wall time: 2.4 seconds

comment:137 Changed 7 months ago by chapoton

I posted it already, but here it is again:

./sage -t --long src/sage/interacts/debugger.py
too many failed tests, not using stored timings
Running doctests with ID 2019-03-21-15-31-58-e3b050c6.
Git branch: python3.7
Using --optional=dochtml,fricas,gfortran,memlimit,mpir,normaliz,python2,sage
Doctesting 1 file.
sage -t --long src/sage/interacts/debugger.py
**********************************************************************
File "src/sage/interacts/debugger.py", line 102, in sage.interacts.debugger.Debug.curframe
Failed example:
    d.curframe()
Expected:
    <frame ...>
Got:
    <repr(<sage.interfaces.sage0.SageElement at 0x7f773c42f870>) failed: TypeError: replace() argument 2 must be str, not SageFunction>
**********************************************************************
1 item had failures:
   1 of   4 in sage.interacts.debugger.Debug.curframe
    [23 tests, 1 failure, 1.38 s]
----------------------------------------------------------------------
sage -t --long src/sage/interacts/debugger.py  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 1.4 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 1.4 seconds

comment:138 Changed 7 months ago by jdemeyer

I have no idea why that's failing for you...

comment:139 Changed 7 months ago by chapoton

Let me try on another machine (will take time)

comment:140 Changed 7 months ago by git

  • Commit changed from eed6a11a07b5efdff43ef711046e07e95fe5a045 to 919ca41c9ad361af2588cc5a25d654269e6cc324

Branch pushed to git repo; I updated commit sha1. New commits:

919ca41parallel/map_reduce.py times out

comment:141 Changed 7 months ago by jdemeyer

This last commit is probably controversial, but it's the best I can do. I have no idea what's going on in map_reduce.py.

comment:142 Changed 7 months ago by jdemeyer

As far as I'm concerned, this is now actually ready for review.

comment:143 follow-ups: Changed 7 months ago by chapoton

Could we instead tag #py2 the doctest that triggers the time-out ?

comment:144 in reply to: ↑ 143 Changed 7 months ago by jdemeyer

Replying to chapoton:

Could we instead tag #py2 the doctest that triggers the time-out ?

I'd rather not do that. For me, # py2 means doctests that are clearly meant to pass only on Python 2; where the fact that it only works on Python 2 is not considered a bug.

comment:145 Changed 7 months ago by chapoton

Then we should fix that bug, and not hide it under the carpet.

comment:146 Changed 7 months ago by jdemeyer

# py2 is hiding it under the carpet. Removing it from python3-known-passing.txt is just deferring it to another time.

comment:147 Changed 7 months ago by chapoton

I get the exact same failure in src/sage/interacts/debugger.py in another machine. So maybe this should not be neglected.

comment:148 Changed 7 months ago by jdemeyer

I believe you, but I cannot reproduce the failure in src/sage/interacts/debugger.py

comment:149 Changed 7 months ago by jhpalmieri

The appropriate tests pass for me on OS X. (I don't see a failure in interacts/debugger.py, either.)

Last edited 7 months ago by jhpalmieri (previous) (diff)

comment:150 Changed 7 months ago by chapoton

So maybe it's time for Erik to launch the Windows build..

comment:151 Changed 7 months ago by embray

Hopefully I'll have time tomorrow but not sure. Kinda crazy day. Monday otherwise.

comment:152 Changed 7 months ago by embray

Well, it compiled fine at least (which was to be expected since the necessary patches are applied) but I haven't had a chance to run any tests yet.

comment:153 Changed 7 months ago by jdemeyer

If you agree to deprecate the whole module src/sage/interacts/debugger.py in #27531, then we could easily mark the failing test as # py2 and not bother with it anymore.

Last edited 7 months ago by jdemeyer (previous) (diff)

comment:154 in reply to: ↑ 143 Changed 7 months ago by jdemeyer

Replying to chapoton:

Could we instead tag #py2 the doctest that triggers the time-out ?

Your suggestion of checking which specific doctest triggers the time-out is a good one. I'll try that now.

comment:155 Changed 7 months ago by git

  • Commit changed from 919ca41c9ad361af2588cc5a25d654269e6cc324 to 2bf16e852f564ab6f351eff364e739499216bf0b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e646670Mark a test for deprecated functionality as Python 2 only
2bf16e8Mark a few tests as known bug (#27537)

comment:156 Changed 7 months ago by jdemeyer

New attempt, hopefully more acceptable. Still needs to be tested.

comment:157 follow-up: Changed 7 months ago by jdemeyer

This passes make ptest-python3 for me.

comment:158 in reply to: ↑ 157 Changed 7 months ago by chapoton

Replying to jdemeyer:

This passes make ptest-python3 for me.

Same here.

comment:159 Changed 7 months ago by jhpalmieri

I used the updated version of src/ext/doctest/python3-known-passing.txt from #27519, and I also added "--long" to the appropriate line of Makefile (why don't we have this option already?). Tests passed.

comment:160 Changed 7 months ago by jdemeyer

So, is anybody willing to give positive review here?

comment:161 Changed 7 months ago by chapoton

Should we wait for Erik's green light ?

comment:162 Changed 7 months ago by embray

Came home after a weekend afk to find that although python3 succeeded in building, the build stopped at cysignals with a strange problem. I think there might have been something else odd going on with my build though so I'll start it again and then run the tests.

comment:163 Changed 7 months ago by embray

  • Milestone changed from sage-8.7 to sage-8.8

Ticket retargeted after milestone closed (if you don't believe this ticket is appropriate for the Sage 8.8 release please retarget manually)

comment:164 Changed 7 months ago by embray

Seeing lots of

[dochtml] [misc     ] loading pickled environment... failed: 'str' object has no attribute '__dict__'

when building the docs. Probably not directly related to this ticket, but worth making note of.

comment:165 Changed 7 months ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_review to needs_work

Python 3.7.3 was just released, so we might as well upgrade to that.

comment:166 Changed 7 months ago by git

  • Commit changed from 2bf16e852f564ab6f351eff364e739499216bf0b to 5a09cf14a56f4da514b40b7734327eb4ea33c16c

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

9d2c0c4Upgrade to Python 3.7.3
438c3f4Disable collections.abc warnings
e62ddf8Fix "abort pool operation" docbuild doctest
0ad1dc1Mark a test for deprecated functionality as Python 2 only
5a09cf1Mark a few tests as known bug (#27537)

comment:167 Changed 7 months ago by embray

Cool, I'll try it.

comment:168 Changed 7 months ago by jdemeyer

  • Status changed from needs_work to needs_review

comment:169 Changed 7 months ago by jdemeyer

3.7.3 allowed me to remove one more Cygwin patch (the one related to the .exe suffix)

comment:170 Changed 7 months ago by embray

Is there any reason its version is "3.7.3.p0" and not just "3.7.3"? Just because we do still apply some patches? Not that I think it matters much, just curious.

comment:171 Changed 7 months ago by jdemeyer

I thought the rule was to use .pX when there are patches. But it's certainly not enforced strictly.

comment:172 Changed 7 months ago by embray

I think that makes sense--after all it is no longer "purely" the 3.7.3 release at that point.

comment:173 Changed 7 months ago by jdemeyer

I'm getting docbuild failures due to

WARNING: cannot copy static file OSError(40, 'Too many levels of symbolic links')

comment:174 Changed 7 months ago by jhpalmieri

I don't see this warning. You don't have that stupid mathjax recursive link, do you? Oh, make ptest-python3 passes for me.

Last edited 7 months ago by jhpalmieri (previous) (diff)

comment:175 Changed 7 months ago by jdemeyer

I don't know what was wrong with the docs. I did make doc-clean and then it worked.

comment:176 Changed 7 months ago by embray

I am having a bad problem when running the tests where every test outputs something like:

Exception ignored in: <function _TemporaryFileCloser.__del__ at 0x6ffff4d92f0>
Traceback (most recent call last):
  File "/home/embray/src/sagemath/sage-python3/local/lib/python3.7/tempfile.py", line 448, in __del__
    self.close()
  File "/home/embray/src/sagemath/sage-python3/local/lib/python3.7/tempfile.py", line 444, in close
    unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpw4zt6tfq'

even if the test succeeds. I think this may be a bug (big shock) in the tempfile module itself. I believe it might even be one that I already have a patch for (I seem to recall fixing some issues in tempfile back when I was trying to get all of the Python test suite to pass on Cygwin) but I will need to look into that before saying anything more.

comment:177 Changed 7 months ago by embray

(Obviously the bug itself is relatively harmless: It's just running a __del__ method that unlinks a file but doesn't ignore errors that occur if the file was already deleted. It's more a nuisance than anything).

comment:178 Changed 7 months ago by embray

I see; this is a recurrence of the problem I originally described here: https://trac.sagemath.org/ticket/25107#comment:14

It's resurfaced, because the tempfile module has been refactored, now in such a way that merely setting the _TemporaryFileWrapper.deleted attribute no longer has any effect whatsoever (it should really probably be a property which delegates to the new _TemporaryFileCloser object.

I think this may be a broader issue with NamedTemporaryFile that is not unique just to Cygwin: If you create a NamedTemporaryFile(delete=True) in a parent process, then add it as an attribute on a Process subclass, when the worker process exits the file is deleted. But then when the temporary file is closed on the parent side it tries to re-delete the already deleted file. I think more generally it would be best if ENOENT errors were just ignored in this case.

comment:179 Changed 7 months ago by embray

If you would consider including this commit, or something like it, that would resolve the problem I'm having with unhandled exceptions in _TemporaryFileCloser.__del__ .

With this fix, I can see more clearly that there are two test failures remaining on Cygwin:

sage -t --long src/sage_setup/docbuild/__init__.py  # 1 doctest failed
sage -t --long src/sage/manifolds/differentiable/vectorfield.py  # Killed due to segmentation fault

The first one is merely a manifestation of #27514. Still need to rethink exactly how that test is implemented.

The second is more disconcerting, being a segfault (but at least repeatable). I will look into it next.

comment:180 Changed 7 months ago by embray

  • Reviewers changed from Vincent Klein to Vincent Klein, Erik Bray
  • Status changed from needs_review to needs_work

comment:181 follow-up: Changed 7 months ago by embray

I think the segfault might be related to openblas issues that I've already fixed, but may not be incorporated in my current build (since I started building this Python 3 branch some time ago, possibly before all the relevant openblas fixes were incorporated).

comment:182 Changed 7 months ago by chapoton

  • Dependencies #27523 deleted

comment:183 follow-up: Changed 7 months ago by chapoton

  • Reviewers changed from Vincent Klein, Erik Bray to Vincent Klein, Erik Bray, Frédéric Chapoton
  • Status changed from needs_work to positive_review

I am setting this to positive now, so that we can start again making progress on #27519. We can always fix details in later tickets.

comment:184 Changed 7 months ago by vbraun

  • Branch changed from u/jdemeyer/upgrade_to_python_3_7_0 to 5a09cf14a56f4da514b40b7734327eb4ea33c16c
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:185 in reply to: ↑ 183 Changed 7 months ago by embray

  • Commit 5a09cf14a56f4da514b40b7734327eb4ea33c16c deleted

Replying to chapoton:

I am setting this to positive now, so that we can start again making progress on #27519. We can always fix details in later tickets.

Why would you do that? I explicitly pointed out that something was broken and proposed a fix, but that fix wasn't applied yet. Doing this "now" has no direct impact on making progress on #27519. You're just rushing things for no reason and leaving them broken.

comment:186 in reply to: ↑ 181 Changed 7 months ago by embray

Replying to embray:

I think the segfault might be related to openblas issues that I've already fixed, but may not be incorporated in my current build (since I started building this Python 3 branch some time ago, possibly before all the relevant openblas fixes were incorporated).

I meant to follow up here. The segfault was nothing to do with Python 3 (fortunately) and is fixed by #27565 . Not exactly sure why I wasn't seeing that bug before--it's possible I was building OpenBLAS out of its recent develop branch which already had that fixed.

comment:187 Changed 6 months ago by fbissey

Do we have a follow up on that ticket anywhere? In sage-on-gentoo I have a segfault when building the documentation with python 3.7.3 (but not 2.7 or 3.6). Am I alone with that issue?

python3.7: /dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/build/cythonized/sage/matrix/matrix1.c:14275: __pyx_pf_4sage_6matrix_7matrix1_6Matrix_60matrix_from_columns: Assertion `PyList_Check(__pyx_v_columns)' failed.

comment:188 Changed 6 months ago by jdemeyer

Interesting. You're building with assertions enabled.

comment:189 Changed 6 months ago by jdemeyer

Can you attach or send me that matrix1.c file? In my version of that file, I don't find such an assertion.

Last edited 6 months ago by jdemeyer (previous) (diff)

Changed 6 months ago by fbissey

comment:190 Changed 6 months ago by fbissey

Attached as requested (compressed).

comment:191 Changed 6 months ago by jdemeyer

Thanks, that clears things up. Follow-up ticket at #27688.

Note: See TracTickets for help on using tickets.