spkg-configure.m4 for python3

[dimpase]

Do not install python3 if a sufficiently new version is available in $PATH. This could be a system python3, or the python3 from within a venv.

As suggested in #29032, we make $SAGE_LOCAL a venv (​https://docs.python.org/3/library/venv.html) over the system python3 -- if a suitable system python3 is found -- by using the venv.EnvBuilder API.

This is done by the new script build/bin/sage-venv, which we use both during configure (to make sure that we select a system python3 for which venvs work correctly) and during make.

The venv does not include the system site-packages: We continue to install all Python packages into our venv using the existing build infrastructure. We keep the task of using system site-packages for the follow-up ticket #29023.

(Note, we use venv (new since Python 3.3), not virtualenv. So this change is limited to Python 3 builds of Sage.)

[embray]

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

[embray]

For Python packages when installing using the system Python we definitely need a well thought out approach, likely with different options for handling questions such as where to install additional Python packages.

$SAGE_LOCAL already acts as a sort-of virtualenv, but we might still consider using virtualenv/venv in $SAGE_LOCAL (with the --system-site-packages option) as a way of extending sys.path while still using as many system Python packages as possible.

[mkoeppe]

See #29013 - "In a python3 build, install all Python packages into a venv"

[mkoeppe]

Narrowed the ticket to py3.

[mkoeppe]

Here's a version that works. The venv is created in build/make/Makefile target (in #29013, I created the venv in the spkg-postinst script of the python3 spkg.)

---

New commits:

​1363425	In a python3 build, install all Python packages into a venv
​a394268	sage-pip-install: Use PYTHON=sage-python23
​a84815b	spkg-configure.m4 for python3

[dimpase]

how about checking for Py3 deps: sqlite libpng bzip2 xz libffi ? (I skipped in this list ones that are 2nd order, so no need to check them)

[dimpase]

On Gentoo Python 3 does not have sqlite module by default. I had to do $ pip install pysqlite3 --user, and still elliptic_curves spkg installation errors out.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0d0495f

build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​98dff9d

Remove use of --system-site-packages

[mkoeppe]

I've reduced the scope of the ticket, excluding the use of system site packages.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6180132

build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module

[dimpase]

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

[mkoeppe]

Thanks for testing!

[dimpase]

Replying to mkoeppe:

I've reduced the scope of the ticket, excluding the use of system site packages.

do you mean pip/etc-installed site-packages of system's Python?

[mkoeppe]

Replying to dimpase:

Replying to mkoeppe:

I've reduced the scope of the ticket, excluding the use of system site packages.

do you mean pip/etc-installed site-packages of system's Python?

Yes.

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​61d7822	Trac #29002: lower the version bound on the system sqlite check.
​3ab4515	Merge branch 'u/mjo/ticket/29002' of git://trac.sagemath.org/sage into t/27824/public/27824-python3-spkg-configure
​c7b9ae4	Add sqlite depcheck
​961f850	Trac #29002: lower the version bound on the system sqlite check (again).
​d321948	Trac #29002: drop the readline dependency check for system sqlite.
​14d15fb	Merge branch 'u/mjo/ticket/29002' of git://trac.sagemath.org/sage into t/27824/public/27824-python3-spkg-configure
​258d112	build/pkgs/pillow/patches/setup.py.patch: Don't fail if Py_MACOS_SYSROOT sysconfig variable does not exist
​43673c0	Merge branch 't/29019/make_patch_to_pillow_more_robust__do_not_depend_on__py_macos_sysroot__' into t/27824/public/27824-python3-spkg-configure
​56b1618	build/pkgs/pillow/spkg-install: Help setup.py find zlib.h on macOS; show more build output
​5930b4e	Merge branch 't/29019/make_patch_to_pillow_more_robust__do_not_depend_on__py_macos_sysroot__' into t/27824/public/27824-python3-spkg-configure

[mkoeppe]

Now this is giving

[pillow-5.3.0.p0] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -g -DHAVE_OPENJPEG -DHAVE_LIBZ -DHAVE_LIBTIFF -DPILLOW_VERSION="5.3.0" -I/usr/local/Cellar/openjpeg/2.3.1/include/openjpeg-2.3 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/var/tmp/sage/build/pillow-5.3.0.p0/src/src/libImaging -I/usr/local/Cellar/jpeg/9c/include -I/usr/local/Cellar/libtiff/4.1.0/include -I/usr/local/Cellar/freetype/2.10.1/include/freetype2 -I/usr/local/Cellar/little-cms2/2.9/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/lib/sage/venv/sage/include -I/usr/local/include -I/usr/local/Cellar/freetype/2.10.1/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/lib/sage/venv/sage/include -I/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/_imaging.c -o build/temp.macosx-10.15-x86_64-3.7/src/_imaging.o
[pillow-5.3.0.p0] error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "10.15" during configure
[pillow-5.3.0.p0] ********************************************************************************

This is coming from the fake MACOSX_DEPLOYMENT_TARGET set in sage-env. See #18272, #7095, #18254, #16312.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3b34e2e	src/bin/sage-env: Don't set MACOSX_DEPLOYMENT_TARGET if we use system python3
​7bd8746	_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL

[mkoeppe]

Replying to dimpase:

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

Fixed now.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d4fbe40	_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
​bba08ec	Create module src/sage/env_config.py from src/sage/env_config.py.in, defining SAGE_LOCAL
​d1779cc	Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
​cd864e8	Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

[mkoeppe]

Compiles OK on macOS with homebrew. make ptestlong mostly good except for

• GLPK output (spkg-configure should really reject a libglpk that talks too much) - it makes it hard to see actual failures
• and numerical failures in src/sage/matrix/matrix2.pyx, src/sage/finance/time_series.pyx, and src/sage/schemes/affine/affine_homset.py as reported in #29013

  sage -t --long src/sage/matrix/matrix2.pyx  # 2 doctests failed
  sage -t --long src/sage/finance/time_series.pyx  # 5 doctests failed
  sage -t --long src/sage/schemes/affine/affine_homset.py  # 1 doctest failed
  

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​85147f9	build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg
​93bae6f	Merge branch 't/29025/numpy__site_cfg_needs_a__default__section' into t/29013/public/29013-use-py3-venv
​669b1df	Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

[dimpase]

Replying to mkoeppe:

Replying to dimpase:

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

Fixed now.

after this fix, it builds and passes most tests on Gentoo (some errors related to cryptominisat that I had installed, and it probably got broken in some way)

[dimpase]

I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)

sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py
**********************************************************************
File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat
Failed example:
    solver = CryptoMiniSat()                                  # optional - cryptominisat
Exception raised:
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__
        from pycryptosat import Solver
    ModuleNotFoundError: No module named 'pycryptosat'

I wonder whether it uses a convoluted way to create python extensions, which got broken.

[dimpase]

I see (on a system where cryptominisat package works) that it is one of few packages that get installed without a sub-directory named the same as package in site-packages:

sage: import pycryptosat
sage: print(pycryptosat.__file__)
/home/scratch2/dimpase/sage/sage/local/lib/python3.7/site-packages/pycryptosat.cpython-37m-x86_64-linux-gnu.so
sage: import numpy
sage: print(numpy.__file__)
/home/scratch2/dimpase/sage/sage/local/lib/python3.7/site-packages/numpy/__init__.py

Not sure whether this is a bug on the package, or a bug in this branch. On this branch I see (with local=SAGE_LOCAL)

$ ls local/lib/python3.7/site-packages/*
local/lib/python3.7/site-packages/pycryptosat-0.2.0-py3.7.egg-info  local/lib/python3.7/site-packages/pycryptosat.cpython-37m-x86_64-linux-gnu.so

So it seems that this branch does not pick up anything from that location.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2ed5d07	Add cryptominisat patch for virtual envs
​61a9e37	Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​27d945a	moved patch to the right place
​8c546cf	Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

[embray]

What's the deal with config_venv? Why is any of that necessary? I looked at the commit that added it but it wasn't clear. It seems to me another reason to avoid all the venv stuff...

[embray]

So, all that $VIRTUAL_ENV/bin/activate does, essentially, is to set 3 environment variables: PATH (to which it inserts $VIRTUAL_ENV/bin in the front), PS1 (which Sage also already sets when activating the sage environment), and the $VIRTUAL_ENV environment variable itself.

Other than that, the only thing that really makes a directory a virtualenv (with Python 3 + venv, I'm not counting the old virtualenv) is the presence of pyvenv.cfg in the install prefix. The format of this file is more-or-less documented (albeit not very clearly) in ​https://docs.python.org/3/library/venv.html. So all we really need to do to turn $SAGE_LOCAL into a virtualenv of a system Python 3 is to write the appropriate pyvenv.cfg into $SAGE_LOCAL, and not mess around with the full venv command at all. That, and make $SAGE_LOCAL/bin/python3 a symlink to the real python3.

Although not strictly necessary, it is also possibly helpful to third-party tools to set the $VIRTUAL_ENV environment variable.

[mkoeppe]

Replying to embray:

What's the deal with config_venv? Why is any of that necessary? I looked at the commit that added it but it wasn't clear.

It is used in a configure test, build/pkgs/python3/spkg-configure.m4 to make sure that the system python has all optional modules that we need.

[mkoeppe]

Replying to embray:

So all we really need to do to turn $SAGE_LOCAL into a virtualenv of a system Python 3 is to write the appropriate pyvenv.cfg into $SAGE_LOCAL, and not mess around with the full venv command at all.

This amounts to trying to upgrade python3's venv to something like a conda environment. This is what I avoid in #29013 on purpose - the venv is only used in a clean, documented way, using the documented tools.

[dimpase]

writing pyvenv.cfg by hand is bound to be broken at some point, as it's an implementation detail of venv. The correct interface to venv is to use venv, I think.

(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))

[mkoeppe]

Replying to dimpase:

(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))

Glad we have this on record now for the software historians/archaeologists to discover :)

[embray]

Replying to dimpase:

writing pyvenv.cfg by hand is bound to be broken at some point, as it's an implementation detail of venv. The correct interface to venv is to use venv, I think.

That's not true. It's documented both in the documentation for the venv module and the site module. If you really want to avoid writing it by hand the venv module includes code for that as well. This (with the system Python 3) is effectively the same (albeit more opaque). From within SAGE_ROOT, but without the sage environment activated:

$ python3 -c 'import venv; b = venv.EnvBuilder(); c = b.ensure_directories("local"); b.create_configuration(c)'
$ cat local/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.6.8

but I think even that's overkill for now.

[embray]

Replying to mkoeppe:

Replying to embray:

What's the deal with config_venv? Why is any of that necessary? I looked at the commit that added it but it wasn't clear.

It is used in a configure test, build/pkgs/python3/spkg-configure.m4 to make sure that the system python has all optional modules that we need.

But why do you need to make a virtualenv for that? Why not run the system python directly?

[mkoeppe]

Replying to embray:

But why do you need to make a virtualenv for that? Why not run the system python directly?

On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.

[mkoeppe]

I'll add this documentation to the branch.

[dimpase]

Replying to mkoeppe:

Replying to dimpase:

(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))

Glad we have this on record now for the software historians/archaeologists to discover :)

it's here: ​https://trac.sagemath.org/ticket/27824?version=0

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​348cf22

build/pkgs/python3/spkg-configure.m4: Add comments

[mkoeppe]

By the way, should I be checking for the ssl module too?

[embray]

Replying to mkoeppe:

Replying to embray:

But why do you need to make a virtualenv for that? Why not run the system python directly?

On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.

But why?

[embray]

Replying to embray:

Replying to mkoeppe:

Replying to embray:

But why do you need to make a virtualenv for that? Why not run the system python directly?

On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.

But why?

Wait, I misunderstood what you wrote at first. I know why it uses a virtualenv. What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

[mkoeppe]

Replying to embray:

What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

Because system python could have a module named sqlite3 in its site-packages; but then when we try to use it in a venv, it won't be there.

[mkoeppe]

Replying to mkoeppe:

By the way, should I be checking for the ssl module too?

I think I'll check for all modules mentioned in #27705.

[embray]

Replying to mkoeppe:

Replying to embray:

What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

Because system python could have a module named sqlite3 in its site-packages; but then when we try to use it in a venv, it won't be there.

So use python3 -S if you really want to be sure. That disables site-packages.

[embray]

An interesting discovery while playing around with this: By default Ubuntu's Python does not have distutils. You have to explicitly apt-get install python3-distutils. So this is something we also need to check for at configure time or else it wont be possible to install Python packages.

[embray]

FWIW I'm experimenting with an alternative solution based on my comments here (albeit with significant pieces taken from this ticket!) I'll update when I have something that I think works...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cb399a1

Set PYTHON_FOR_VENV correctly, use it in build/make/deps

[mkoeppe]

Replying to embray:

By default Ubuntu's Python does not have distutils. You have to explicitly apt-get install python3-distutils. So this is something we also need to check for at configure time or else it wont be possible to install Python packages.

Thanks! I'll add this test.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​71cc491

build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core

[mkoeppe]

Also, of course, one needs the Python headers, on Ubuntu from libpython3.7-dev.

[mkoeppe]

So I'll add a check that the distutils can actually build an extension.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c7c1671

build/pkgs/python3/spkg-configure.m4: Check that distutils can build an extension

[mkoeppe]

Testing with the attached Dockerfile on ubuntu bionic.

module and distutils checking seems to work correctly.

Got stuck at scipy install.

[scipy-1.2.0]     numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

Looks like scipy would need a sitecfg as well so that it finds BLAS. It's strange that we build numpy with providing site.cfg (which we just had to correct in #29025), but scipy without site.cfg.

[mkoeppe]

Replying to mkoeppe:

Testing with the attached Dockerfile on ubuntu bionic.

module and distutils checking seems to work correctly.

Got stuck at scipy install.

[scipy-1.2.0]     numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

Looks like scipy would need a sitecfg as well so that it finds BLAS. It's strange that we build numpy with providing site.cfg (which we just had to correct in #29025), but scipy without site.cfg.

@fbissey, based on your work on scipy configuration in #20157, would you be able to help with this?

[fbissey]

You could provide an identical site.cfg for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.

[dimpase]

Replying to mkoeppe:

Replying to embray:

What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

Because system python could have a module named sqlite3 in its site-packages; but then when we try to use it in a venv, it won't be there.

I wonder about rationale for disabling system's site-packages. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages)?

[mkoeppe]

I wonder about rationale for disabling system's site-packages. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages)?

It’s strictly for making this ticket easier because I don’t have to worry about getting the correct versions of all Python packages. See #29023 for the next step - let’s discuss any issues regarding use of system site packages there.

[embray]

Replying to mkoeppe:

I wonder about rationale for disabling system's site-packages. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages)?

It’s strictly for making this ticket easier because I don’t have to worry about getting the correct versions of all Python packages. See #29023 for the next step - let’s discuss any issues regarding use of system site packages there.

I agree here, it's much simpler for now to focus on using a "bare" Python as much as possible. I think detecting and using system Python packages is where we want to go, but it can get a bit hairier...

[embray]

For the heck of it, I'm seeing how far I can get with a system Python 3.6. I think anything less than 3.6 is almost definitely too old, but I have a feeling 3.6 might still be usable as a minimum. The only major hassle I've had so far is ensuring that it uses UTF-8 encoding by default.

[mkoeppe]

Replying to fbissey:

You could provide an identical site.cfg for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.

Thank you! The file is installed. I'll investigate why it is not enough.

[embray]

Replying to embray:

FWIW I'm experimenting with an alternative solution based on my comments here (albeit with significant pieces taken from this ticket!) I'll update when I have something that I think works...

Here's a prototype of my alternative approach: #29032

I admit it does not yet address as many issues as this ticket does, like all the site.cfg fixes with Numpy. Personally I didn't need it, but if those fixes are necessary they can easily be applied to my ticket as well.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​356b6d7

build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"

[mkoeppe]

(extracted from Erik's commit at #29032.)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​92953ce

build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​08ec551	build/make/deps: Make python3_venv a .PHONY target
​b2ab6f3	src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
​7a43544	src/sage/env.py, envconfig.py.in: Add MAXIMA; src/sage/interfaces/maxima.py: Use it
​f2af478	Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
​8959c19	Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure
​4c78bdf	src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032

[mkoeppe]

Replying to mkoeppe:

Replying to fbissey:

You could provide an identical site.cfg for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.

Thank you! The file is installed. I'll investigate why it is not enough.

I've created ticket #29051 for this (Paths configured in installed numpy site.cfg [DEFAULT/ALL] do not affect scipy)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​56e3daa	src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
​208c7e6	src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL
​1bd8aef	spkg-configure.m4 for python3
​7237aa4	build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi
​d044744	build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module
​c6fec13	src/bin/sage-env: Don't set MACOSX_DEPLOYMENT_TARGET if we use system python3
​b69ac5b	Set PYTHON_FOR_VENV correctly, use it in build/make/deps
​c143043	build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
​973c36f	build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
​3d90535	src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​73e730c	build/bin/sage-system-python: Prefer python3, check a larger list of executable names, check for urllib; exit with a proper error message if no suitable Python is found
​b7c5335	configure.ac: Stop early if no suitable Python is found

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3aa209e

src/bin/sage-location: Use sage-python instead of sage-system-python

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

[mkoeppe]

(pushed to wrong ticket, corrected)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a7cfbab	build/pkgs/python3/spkg-configure.m4: Rename config_venv -> conftest_venv
​dfa2816	Makefile [misc-clean): No need to remove config_venv

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8f44123

src/bin/sage-env, src/bin/sage-env-config.in: export/use environment variable PYTHON_FOR_VENV

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​c0a99e5	Set PYTHON_FOR_VENV correctly, use it in build/make/deps
​6fcd7cd	build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
​15eceae	build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
​fde251b	src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032
​e73a428	build/pkgs/python3/spkg-configure.m4: Rename config_venv -> conftest_venv
​b1518b7	Makefile [misc-clean): No need to remove config_venv
​179d2e5	src/bin/sage-env, src/bin/sage-env-config.in: export/use environment variable PYTHON_FOR_VENV
​176ce61	build/make/Makefile.in, build/make/deps: Create venv in $SAGE_LOCAL using venv.EnvBuilder, instead of in $SAGE_LOCAL/lib/sage/venv using venv command line
​e29087a	build/pkgs/python3/spkg-configure.m4: Only set PYTHON_FOR_VENV if we use the system python
​f7924eb	build/bin/sage-venv: New, use it in build/pkgs/python3/spkg-configure.m4 and build/make/deps

[mkoeppe]

This ticket is no longer on top of #29013.

[mkoeppe]

Tests run at ​https://github.com/mkoeppe/sage/actions/runs/58992626 and ​https://github.com/mkoeppe/sage/actions/runs/58992623

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c570d6b

build/bin/sage-venv: Fixup

[mkoeppe]

A successful test at: ​https://github.com/mkoeppe/sage/runs/520449412

[mkoeppe]

Tests run at ​https://github.com/mkoeppe/sage/actions/runs/60588138

[mkoeppe]

Breaks python 2 builds

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​65a9f02

build/make/deps: Do not set PYTHON = python3_venv in a python2 build

[mkoeppe]

Ready for review

[mkoeppe]

New tests at ​https://github.com/mkoeppe/sage/actions/runs/60877259; see also #29367

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1c845a4

build/pkgs/python3/spkg-configure.m4: Do not use venv for python3 if we do a python2 build

[mkoeppe]

Tests at ​https://github.com/mkoeppe/sage/actions/runs/61077298

[dimpase]

Let's move this, looks good to me.

[mkoeppe]

Thanks!

[mmezzarobba]

Hello everyone,

When trying to upgrade to 9.1.rc5, I'm seeing an issue that I suspect could be a side effect of this ticket.

For some reason my sage/local/bin/python3.7 was a symlink to the system python binary. Sage decided it wanted to install its own python-3.7.3.p1, but that failed with a permission error on the symlink. After deleting the symlink, python builds and installs correctly. (I let the logs be overwritten, so I won't be able to provide much details — sorry.)

The overall Sage build still fails with an error about six not being found while building setuptools that I haven't yet investigated.

[dimpase]

if you switch to py3 from py2, you probably need to run ./configure --with-python=3

it is not clear what version you were on, but the changes to the build system were very substantial, so ymmv.

[mkoeppe]

Yes, this is a known limitation. I have created #29708 (Fix switching between --with-system-python3 and --without-system-python3) for it. We shall fix it in 9.2.