Opened 19 months ago

Closed 8 months ago

Last modified 6 months ago

#27824 closed enhancement (fixed)

spkg-configure.m4 for python3

Reported by: dimpase Owned by:
Priority: major Milestone: sage-9.1
Component: build: configure Keywords:
Cc: jdemeyer, embray, vbraun, fbissey, jhpalmieri, vdelecroix Merged in:
Authors: Matthias Koeppe, Erik Bray Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 1c845a4 (Commits) Commit:
Dependencies: Stopgaps:

Description (last modified by mkoeppe)

Do not install python3 if a sufficiently new version is available in $PATH. This could be a system python3, or the python3 from within a venv.

As suggested in #29032, we make $SAGE_LOCAL a venv (https://docs.python.org/3/library/venv.html) over the system python3 -- if a suitable system python3 is found -- by using the venv.EnvBuilder API.

This is done by the new script build/bin/sage-venv, which we use both during configure (to make sure that we select a system python3 for which venvs work correctly) and during make.

The venv does not include the system site-packages: We continue to install all Python packages into our venv using the existing build infrastructure. We keep the task of using system site-packages for the follow-up ticket #29023.

(Note, we use venv (new since Python 3.3), not virtualenv. So this change is limited to Python 3 builds of Sage.)

Attachments (1)

Dockerfile-ubuntu-minimal (3.8 KB) - added by mkoeppe 10 months ago.

Download all attachments as: .zip

Change History (106)

comment:1 Changed 19 months ago by dimpase

  • Description modified (diff)

comment:2 Changed 19 months ago by dimpase

  • Description modified (diff)

comment:3 Changed 18 months ago by embray

  • Milestone sage-8.8 deleted

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

comment:4 Changed 16 months ago by embray

For Python packages when installing using the system Python we definitely need a well thought out approach, likely with different options for handling questions such as where to install additional Python packages.

$SAGE_LOCAL already acts as a sort-of virtualenv, but we might still consider using virtualenv/venv in $SAGE_LOCAL (with the --system-site-packages option) as a way of extending sys.path while still using as many system Python packages as possible.

comment:5 Changed 10 months ago by mkoeppe

See #29013 - "In a python3 build, install all Python packages into a venv"

comment:6 Changed 10 months ago by mkoeppe

  • Dependencies set to #29013
  • Description modified (diff)
  • Summary changed from spkg-configure.m4 for pythons to spkg-configure.m4 for python3

Narrowed the ticket to py3.

comment:7 Changed 10 months ago by mkoeppe

  • Branch set to public/27824-python3-spkg-configure

comment:8 Changed 10 months ago by mkoeppe

  • Commit set to a84815b72ab41f6d0d17b733f44e4e8ec05d3410

Here's a version that works. The venv is created in build/make/Makefile target (in #29013, I created the venv in the spkg-postinst script of the python3 spkg.)


New commits:

1363425In a python3 build, install all Python packages into a venv
a394268sage-pip-install: Use PYTHON=sage-python23
a84815bspkg-configure.m4 for python3

comment:9 Changed 10 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Milestone set to sage-9.2

comment:10 Changed 10 months ago by dimpase

how about checking for Py3 deps: sqlite libpng bzip2 xz libffi ? (I skipped in this list ones that are 2nd order, so no need to check them)

comment:11 Changed 10 months ago by dimpase

On Gentoo Python 3 does not have sqlite module by default. I had to do $ pip install pysqlite3 --user, and still elliptic_curves spkg installation errors out.

comment:12 Changed 10 months ago by git

  • Commit changed from a84815b72ab41f6d0d17b733f44e4e8ec05d3410 to 0d0495fd61f789981d3074ae142b89e248ee394f

Branch pushed to git repo; I updated commit sha1. New commits:

0d0495fbuild/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi

comment:13 Changed 10 months ago by git

  • Commit changed from 0d0495fd61f789981d3074ae142b89e248ee394f to 98dff9d32325d99c447132a28c4eae6c2d242d9a

Branch pushed to git repo; I updated commit sha1. New commits:

98dff9dRemove use of --system-site-packages

comment:14 follow-up: Changed 10 months ago by mkoeppe

  • Cc jdemeyer embray vbraun added
  • Description modified (diff)

I've reduced the scope of the ticket, excluding the use of system site packages.

Last edited 10 months ago by mkoeppe (previous) (diff)

comment:15 Changed 10 months ago by git

  • Commit changed from 98dff9d32325d99c447132a28c4eae6c2d242d9a to 61801321f81917550a065462422bb2a209bc1030

Branch pushed to git repo; I updated commit sha1. New commits:

6180132build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module

comment:16 follow-up: Changed 10 months ago by dimpase

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

comment:17 Changed 10 months ago by mkoeppe

Thanks for testing!

comment:18 in reply to: ↑ 14 ; follow-up: Changed 10 months ago by dimpase

Replying to mkoeppe:

I've reduced the scope of the ticket, excluding the use of system site packages.

do you mean pip/etc-installed site-packages of system's Python?

comment:19 in reply to: ↑ 18 Changed 10 months ago by mkoeppe

Replying to dimpase:

Replying to mkoeppe:

I've reduced the scope of the ticket, excluding the use of system site packages.

do you mean pip/etc-installed site-packages of system's Python?

Yes.

comment:20 Changed 10 months ago by mkoeppe

  • Dependencies changed from #29013 to #29013, #29019

comment:21 Changed 10 months ago by git

  • Commit changed from 61801321f81917550a065462422bb2a209bc1030 to 5930b4e219e3f12fb675ba2caf5c1d7fcb1b15e4

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

61d7822Trac #29002: lower the version bound on the system sqlite check.
3ab4515Merge branch 'u/mjo/ticket/29002' of git://trac.sagemath.org/sage into t/27824/public/27824-python3-spkg-configure
c7b9ae4Add sqlite depcheck
961f850Trac #29002: lower the version bound on the system sqlite check (again).
d321948Trac #29002: drop the readline dependency check for system sqlite.
14d15fbMerge branch 'u/mjo/ticket/29002' of git://trac.sagemath.org/sage into t/27824/public/27824-python3-spkg-configure
258d112build/pkgs/pillow/patches/setup.py.patch: Don't fail if Py_MACOS_SYSROOT sysconfig variable does not exist
43673c0Merge branch 't/29019/make_patch_to_pillow_more_robust__do_not_depend_on__py_macos_sysroot__' into t/27824/public/27824-python3-spkg-configure
56b1618build/pkgs/pillow/spkg-install: Help setup.py find zlib.h on macOS; show more build output
5930b4eMerge branch 't/29019/make_patch_to_pillow_more_robust__do_not_depend_on__py_macos_sysroot__' into t/27824/public/27824-python3-spkg-configure

comment:22 Changed 10 months ago by mkoeppe

Now this is giving

[pillow-5.3.0.p0] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -g -DHAVE_OPENJPEG -DHAVE_LIBZ -DHAVE_LIBTIFF -DPILLOW_VERSION="5.3.0" -I/usr/local/Cellar/openjpeg/2.3.1/include/openjpeg-2.3 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/var/tmp/sage/build/pillow-5.3.0.p0/src/src/libImaging -I/usr/local/Cellar/jpeg/9c/include -I/usr/local/Cellar/libtiff/4.1.0/include -I/usr/local/Cellar/freetype/2.10.1/include/freetype2 -I/usr/local/Cellar/little-cms2/2.9/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/lib/sage/venv/sage/include -I/usr/local/include -I/usr/local/Cellar/freetype/2.10.1/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/lib/sage/venv/sage/include -I/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/_imaging.c -o build/temp.macosx-10.15-x86_64-3.7/src/_imaging.o
[pillow-5.3.0.p0] error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "10.15" during configure
[pillow-5.3.0.p0] ********************************************************************************

This is coming from the fake MACOSX_DEPLOYMENT_TARGET set in sage-env. See #18272, #7095, #18254, #16312.

comment:23 Changed 10 months ago by git

  • Commit changed from 5930b4e219e3f12fb675ba2caf5c1d7fcb1b15e4 to 7bd87464eb40adfe7f9110a452cd10b3c78b8f4c

Branch pushed to git repo; I updated commit sha1. New commits:

3b34e2esrc/bin/sage-env: Don't set MACOSX_DEPLOYMENT_TARGET if we use system python3
7bd8746_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL

comment:24 in reply to: ↑ 16 ; follow-up: Changed 10 months ago by mkoeppe

Replying to dimpase:

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

Fixed now.

comment:25 Changed 10 months ago by mkoeppe

  • Description modified (diff)

comment:26 Changed 10 months ago by git

  • Commit changed from 7bd87464eb40adfe7f9110a452cd10b3c78b8f4c to cd864e89500858a87f0f82faf4a634d200fe2118

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d4fbe40_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
bba08ecCreate module src/sage/env_config.py from src/sage/env_config.py.in, defining SAGE_LOCAL
d1779ccMerge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
cd864e8Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

comment:27 Changed 10 months ago by mkoeppe

Compiles OK on macOS with homebrew. make ptestlong mostly good except for

  • GLPK output (spkg-configure should really reject a libglpk that talks too much) - it makes it hard to see actual failures
  • and numerical failures in src/sage/matrix/matrix2.pyx, src/sage/finance/time_series.pyx, and src/sage/schemes/affine/affine_homset.py as reported in #29013
    sage -t --long src/sage/matrix/matrix2.pyx  # 2 doctests failed
    sage -t --long src/sage/finance/time_series.pyx  # 5 doctests failed
    sage -t --long src/sage/schemes/affine/affine_homset.py  # 1 doctest failed
    

comment:28 Changed 10 months ago by git

  • Commit changed from cd864e89500858a87f0f82faf4a634d200fe2118 to 669b1dfa8bfd50c0710f546ed2bdfe43b14bc13d

Branch pushed to git repo; I updated commit sha1. New commits:

85147f9build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg
93bae6fMerge branch 't/29025/numpy__site_cfg_needs_a__default__section' into t/29013/public/29013-use-py3-venv
669b1dfMerge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

comment:29 in reply to: ↑ 24 Changed 10 months ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

Fixed now.

after this fix, it builds and passes most tests on Gentoo (some errors related to cryptominisat that I had installed, and it probably got broken in some way)

comment:30 Changed 10 months ago by dimpase

I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)

sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py
**********************************************************************
File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat
Failed example:
    solver = CryptoMiniSat()                                  # optional - cryptominisat
Exception raised:
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__
        from pycryptosat import Solver
    ModuleNotFoundError: No module named 'pycryptosat'

I wonder whether it uses a convoluted way to create python extensions, which got broken.

comment:31 Changed 10 months ago by dimpase

I see (on a system where cryptominisat package works) that it is one of few packages that get installed without a sub-directory named the same as package in site-packages:

sage: import pycryptosat
sage: print(pycryptosat.__file__)
/home/scratch2/dimpase/sage/sage/local/lib/python3.7/site-packages/pycryptosat.cpython-37m-x86_64-linux-gnu.so
sage: import numpy
sage: print(numpy.__file__)
/home/scratch2/dimpase/sage/sage/local/lib/python3.7/site-packages/numpy/__init__.py

Not sure whether this is a bug on the package, or a bug in this branch. On this branch I see (with local=SAGE_LOCAL)

$ ls local/lib/python3.7/site-packages/*
local/lib/python3.7/site-packages/pycryptosat-0.2.0-py3.7.egg-info  local/lib/python3.7/site-packages/pycryptosat.cpython-37m-x86_64-linux-gnu.so

So it seems that this branch does not pick up anything from that location.

comment:32 Changed 10 months ago by git

  • Commit changed from 669b1dfa8bfd50c0710f546ed2bdfe43b14bc13d to 61a9e3733989f265bda6cea7e6bccf7bf88c9749

Branch pushed to git repo; I updated commit sha1. New commits:

2ed5d07Add cryptominisat patch for virtual envs
61a9e37Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

comment:33 Changed 10 months ago by git

  • Commit changed from 61a9e3733989f265bda6cea7e6bccf7bf88c9749 to 8c546cfd4a35c82c911e3620538c2188e7e350f2

Branch pushed to git repo; I updated commit sha1. New commits:

27d945amoved patch to the right place
8c546cfMerge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure

comment:34 follow-up: Changed 10 months ago by embray

What's the deal with config_venv? Why is any of that necessary? I looked at the commit that added it but it wasn't clear. It seems to me another reason to avoid all the venv stuff...

comment:35 follow-up: Changed 10 months ago by embray

So, all that $VIRTUAL_ENV/bin/activate does, essentially, is to set 3 environment variables: PATH (to which it inserts $VIRTUAL_ENV/bin in the front), PS1 (which Sage also already sets when activating the sage environment), and the $VIRTUAL_ENV environment variable itself.

Other than that, the only thing that really makes a directory a virtualenv (with Python 3 + venv, I'm not counting the old virtualenv) is the presence of pyvenv.cfg in the install prefix. The format of this file is more-or-less documented (albeit not very clearly) in https://docs.python.org/3/library/venv.html. So all we really need to do to turn $SAGE_LOCAL into a virtualenv of a system Python 3 is to write the appropriate pyvenv.cfg into $SAGE_LOCAL, and not mess around with the full venv command at all. That, and make $SAGE_LOCAL/bin/python3 a symlink to the real python3.

Although not strictly necessary, it is also possibly helpful to third-party tools to set the $VIRTUAL_ENV environment variable.

Last edited 10 months ago by embray (previous) (diff)

comment:36 in reply to: ↑ 34 ; follow-up: Changed 10 months ago by mkoeppe

Replying to embray:

What's the deal with config_venv? Why is any of that necessary? I looked at the commit that added it but it wasn't clear.

It is used in a configure test, build/pkgs/python3/spkg-configure.m4 to make sure that the system python has all optional modules that we need.

comment:37 in reply to: ↑ 35 Changed 10 months ago by mkoeppe

Replying to embray:

So all we really need to do to turn $SAGE_LOCAL into a virtualenv of a system Python 3 is to write the appropriate pyvenv.cfg into $SAGE_LOCAL, and not mess around with the full venv command at all.

This amounts to trying to upgrade python3's venv to something like a conda environment. This is what I avoid in #29013 on purpose - the venv is only used in a clean, documented way, using the documented tools.

comment:38 follow-ups: Changed 10 months ago by dimpase

writing pyvenv.cfg by hand is bound to be broken at some point, as it's an implementation detail of venv. The correct interface to venv is to use venv, I think.

(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))

Last edited 10 months ago by dimpase (previous) (diff)

comment:39 in reply to: ↑ 38 ; follow-up: Changed 10 months ago by mkoeppe

Replying to dimpase:

(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))

Glad we have this on record now for the software historians/archaeologists to discover :)

comment:40 in reply to: ↑ 38 Changed 10 months ago by embray

Replying to dimpase:

writing pyvenv.cfg by hand is bound to be broken at some point, as it's an implementation detail of venv. The correct interface to venv is to use venv, I think.

That's not true. It's documented both in the documentation for the venv module and the site module. If you really want to avoid writing it by hand the venv module includes code for that as well. This (with the system Python 3) is effectively the same (albeit more opaque). From within SAGE_ROOT, but without the sage environment activated:

$ python3 -c 'import venv; b = venv.EnvBuilder(); c = b.ensure_directories("local"); b.create_configuration(c)'
$ cat local/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.6.8

but I think even that's overkill for now.

comment:41 in reply to: ↑ 36 ; follow-up: Changed 10 months ago by embray

Replying to mkoeppe:

Replying to embray:

What's the deal with config_venv? Why is any of that necessary? I looked at the commit that added it but it wasn't clear.

It is used in a configure test, build/pkgs/python3/spkg-configure.m4 to make sure that the system python has all optional modules that we need.

But why do you need to make a virtualenv for that? Why not run the system python directly?

comment:42 in reply to: ↑ 41 ; follow-up: Changed 10 months ago by mkoeppe

Replying to embray:

But why do you need to make a virtualenv for that? Why not run the system python directly?

On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.

comment:43 Changed 10 months ago by mkoeppe

I'll add this documentation to the branch.

comment:44 in reply to: ↑ 39 Changed 10 months ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))

Glad we have this on record now for the software historians/archaeologists to discover :)

it's here: https://trac.sagemath.org/ticket/27824?version=0

comment:45 Changed 10 months ago by git

  • Commit changed from 8c546cfd4a35c82c911e3620538c2188e7e350f2 to 348cf2243ce7c8696504a5d85eee849c24cef2a9

Branch pushed to git repo; I updated commit sha1. New commits:

348cf22build/pkgs/python3/spkg-configure.m4: Add comments

comment:46 follow-up: Changed 10 months ago by mkoeppe

By the way, should I be checking for the ssl module too?

comment:47 in reply to: ↑ 42 ; follow-up: Changed 10 months ago by embray

Replying to mkoeppe:

Replying to embray:

But why do you need to make a virtualenv for that? Why not run the system python directly?

On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.

But why?

comment:48 in reply to: ↑ 47 ; follow-up: Changed 10 months ago by embray

Replying to embray:

Replying to mkoeppe:

Replying to embray:

But why do you need to make a virtualenv for that? Why not run the system python directly?

On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.

But why?

Wait, I misunderstood what you wrote at first. I know why it uses a virtualenv. What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

comment:49 in reply to: ↑ 48 ; follow-ups: Changed 10 months ago by mkoeppe

Replying to embray:

What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

Because system python could have a module named sqlite3 in its site-packages; but then when we try to use it in a venv, it won't be there.

comment:50 in reply to: ↑ 46 Changed 10 months ago by mkoeppe

Replying to mkoeppe:

By the way, should I be checking for the ssl module too?

I think I'll check for all modules mentioned in #27705.

comment:51 in reply to: ↑ 49 Changed 10 months ago by embray

Replying to mkoeppe:

Replying to embray:

What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

Because system python could have a module named sqlite3 in its site-packages; but then when we try to use it in a venv, it won't be there.

So use python3 -S if you really want to be sure. That disables site-packages.

comment:52 follow-up: Changed 10 months ago by embray

An interesting discovery while playing around with this: By default Ubuntu's Python does not have distutils. You have to explicitly apt-get install python3-distutils. So this is something we also need to check for at configure time or else it wont be possible to install Python packages.

comment:53 follow-up: Changed 10 months ago by embray

FWIW I'm experimenting with an alternative solution based on my comments here (albeit with significant pieces taken from this ticket!) I'll update when I have something that I think works...

comment:54 Changed 10 months ago by git

  • Commit changed from 348cf2243ce7c8696504a5d85eee849c24cef2a9 to cb399a1fcfd4d8a74c90bad95d1e5d7e521f19a1

Branch pushed to git repo; I updated commit sha1. New commits:

cb399a1Set PYTHON_FOR_VENV correctly, use it in build/make/deps

comment:55 in reply to: ↑ 52 Changed 10 months ago by mkoeppe

Replying to embray:

By default Ubuntu's Python does not have distutils. You have to explicitly apt-get install python3-distutils. So this is something we also need to check for at configure time or else it wont be possible to install Python packages.

Thanks! I'll add this test.

comment:56 Changed 10 months ago by git

  • Commit changed from cb399a1fcfd4d8a74c90bad95d1e5d7e521f19a1 to 71cc4919119784d243d65f853f336771e93e4478

Branch pushed to git repo; I updated commit sha1. New commits:

71cc491build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core

comment:57 Changed 10 months ago by mkoeppe

Also, of course, one needs the Python headers, on Ubuntu from libpython3.7-dev.

Last edited 10 months ago by mkoeppe (previous) (diff)

comment:58 Changed 10 months ago by mkoeppe

So I'll add a check that the distutils can actually build an extension.

comment:59 Changed 10 months ago by git

  • Commit changed from 71cc4919119784d243d65f853f336771e93e4478 to c7c16711c2f3ab262c8d76f6b565727bfc01edff

Branch pushed to git repo; I updated commit sha1. New commits:

c7c1671build/pkgs/python3/spkg-configure.m4: Check that distutils can build an extension

Changed 10 months ago by mkoeppe

comment:60 follow-up: Changed 10 months ago by mkoeppe

Testing with the attached Dockerfile on ubuntu bionic.

module and distutils checking seems to work correctly.

Got stuck at scipy install.

[scipy-1.2.0]     numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

Looks like scipy would need a sitecfg as well so that it finds BLAS. It's strange that we build numpy with providing site.cfg (which we just had to correct in #29025), but scipy without site.cfg.

Last edited 10 months ago by mkoeppe (previous) (diff)

comment:61 in reply to: ↑ 60 Changed 10 months ago by mkoeppe

  • Cc fbissey added

Replying to mkoeppe:

Testing with the attached Dockerfile on ubuntu bionic.

module and distutils checking seems to work correctly.

Got stuck at scipy install.

[scipy-1.2.0]     numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.

Looks like scipy would need a sitecfg as well so that it finds BLAS. It's strange that we build numpy with providing site.cfg (which we just had to correct in #29025), but scipy without site.cfg.

@fbissey, based on your work on scipy configuration in #20157, would you be able to help with this?

comment:62 follow-up: Changed 10 months ago by fbissey

You could provide an identical site.cfg for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.

comment:63 in reply to: ↑ 49 ; follow-up: Changed 10 months ago by dimpase

Replying to mkoeppe:

Replying to embray:

What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?

Because system python could have a module named sqlite3 in its site-packages; but then when we try to use it in a venv, it won't be there.

I wonder about rationale for disabling system's site-packages. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages)?

comment:64 in reply to: ↑ 63 ; follow-up: Changed 10 months ago by mkoeppe

I wonder about rationale for disabling system's site-packages. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages)?

It’s strictly for making this ticket easier because I don’t have to worry about getting the correct versions of all Python packages. See #29023 for the next step - let’s discuss any issues regarding use of system site packages there.

comment:65 in reply to: ↑ 64 Changed 10 months ago by embray

Replying to mkoeppe:

I wonder about rationale for disabling system's site-packages. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages)?

It’s strictly for making this ticket easier because I don’t have to worry about getting the correct versions of all Python packages. See #29023 for the next step - let’s discuss any issues regarding use of system site packages there.

I agree here, it's much simpler for now to focus on using a "bare" Python as much as possible. I think detecting and using system Python packages is where we want to go, but it can get a bit hairier...

comment:66 Changed 10 months ago by embray

For the heck of it, I'm seeing how far I can get with a system Python 3.6. I think anything less than 3.6 is almost definitely too old, but I have a feeling 3.6 might still be usable as a minimum. The only major hassle I've had so far is ensuring that it uses UTF-8 encoding by default.

comment:67 Changed 10 months ago by embray

  • Dependencies changed from #29013, #29019 to #29002, #29013, #29019

comment:68 in reply to: ↑ 62 ; follow-up: Changed 10 months ago by mkoeppe

Replying to fbissey:

You could provide an identical site.cfg for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.

Thank you! The file is installed. I'll investigate why it is not enough.

comment:69 in reply to: ↑ 53 Changed 10 months ago by embray

Replying to embray:

FWIW I'm experimenting with an alternative solution based on my comments here (albeit with significant pieces taken from this ticket!) I'll update when I have something that I think works...

Here's a prototype of my alternative approach: #29032

I admit it does not yet address as many issues as this ticket does, like all the site.cfg fixes with Numpy. Personally I didn't need it, but if those fixes are necessary they can easily be applied to my ticket as well.

comment:70 Changed 10 months ago by git

  • Commit changed from c7c16711c2f3ab262c8d76f6b565727bfc01edff to 356b6d78d66202a17ebcace6178baea91373deb3

Branch pushed to git repo; I updated commit sha1. New commits:

356b6d7build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"

comment:71 Changed 10 months ago by mkoeppe

(extracted from Erik's commit at #29032.)

comment:72 Changed 10 months ago by git

  • Commit changed from 356b6d78d66202a17ebcace6178baea91373deb3 to 92953ceda3f7ba7d6b0246fa5828372bf352dc84

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

92953cebuild/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"

comment:73 Changed 10 months ago by mkoeppe

  • Authors changed from Matthias Koeppe to Matthias Koeppe, Erik Bray

comment:74 Changed 10 months ago by git

  • Commit changed from 92953ceda3f7ba7d6b0246fa5828372bf352dc84 to 4c78bdfccf72183181ee666909dbe365f50b7143

Branch pushed to git repo; I updated commit sha1. New commits:

08ec551build/make/deps: Make python3_venv a .PHONY target
b2ab6f3src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
7a43544src/sage/env.py, envconfig.py.in: Add MAXIMA; src/sage/interfaces/maxima.py: Use it
f2af478Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
8959c19Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure
4c78bdfsrc/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032

comment:75 in reply to: ↑ 68 Changed 10 months ago by mkoeppe

Replying to mkoeppe:

Replying to fbissey:

You could provide an identical site.cfg for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.

Thank you! The file is installed. I'll investigate why it is not enough.

I've created ticket #29051 for this (Paths configured in installed numpy site.cfg [DEFAULT/ALL] do not affect scipy)

comment:76 Changed 8 months ago by git

  • Commit changed from 4c78bdfccf72183181ee666909dbe365f50b7143 to 3d90535b3301e49433e98146e2179c6daa549bf2

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

56e3daasrc/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
208c7e6src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL
1bd8aefspkg-configure.m4 for python3
7237aa4build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi
d044744build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module
c6fec13src/bin/sage-env: Don't set MACOSX_DEPLOYMENT_TARGET if we use system python3
b69ac5bSet PYTHON_FOR_VENV correctly, use it in build/make/deps
c143043build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
973c36fbuild/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
3d90535src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032

comment:77 Changed 8 months ago by mkoeppe

  • Dependencies changed from #29002, #29013, #29019 to #29013

comment:78 Changed 8 months ago by git

  • Commit changed from 3d90535b3301e49433e98146e2179c6daa549bf2 to b7c5335886a174712d11b3d24a7b485e4d30a820

Branch pushed to git repo; I updated commit sha1. New commits:

73e730cbuild/bin/sage-system-python: Prefer python3, check a larger list of executable names, check for urllib; exit with a proper error message if no suitable Python is found
b7c5335configure.ac: Stop early if no suitable Python is found

comment:79 Changed 8 months ago by git

  • Commit changed from b7c5335886a174712d11b3d24a7b485e4d30a820 to 3aa209ef8ca3605b78dbdd7fa59b3a337593ac6e

Branch pushed to git repo; I updated commit sha1. New commits:

3aa209esrc/bin/sage-location: Use sage-python instead of sage-system-python

comment:80 Changed 8 months ago by git

  • Commit changed from 3aa209ef8ca3605b78dbdd7fa59b3a337593ac6e to 3d90535b3301e49433e98146e2179c6daa549bf2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

comment:81 Changed 8 months ago by mkoeppe

(pushed to wrong ticket, corrected)

comment:82 Changed 8 months ago by git

  • Commit changed from 3d90535b3301e49433e98146e2179c6daa549bf2 to dfa28169a5a2c20846df1f2f134e513a66091ede

Branch pushed to git repo; I updated commit sha1. New commits:

a7cfbabbuild/pkgs/python3/spkg-configure.m4: Rename config_venv -> conftest_venv
dfa2816Makefile [misc-clean): No need to remove config_venv

comment:83 Changed 8 months ago by git

  • Commit changed from dfa28169a5a2c20846df1f2f134e513a66091ede to 8f441234e408c4a45e3ebe329d8e96c12e0da78e

Branch pushed to git repo; I updated commit sha1. New commits:

8f44123src/bin/sage-env, src/bin/sage-env-config.in: export/use environment variable PYTHON_FOR_VENV

comment:84 Changed 8 months ago by git

  • Commit changed from 8f441234e408c4a45e3ebe329d8e96c12e0da78e to f7924eb3b6140a725a5181c71936fb9ed65ec5db

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

c0a99e5Set PYTHON_FOR_VENV correctly, use it in build/make/deps
6fcd7cdbuild/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
15eceaebuild/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
fde251bsrc/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032
e73a428build/pkgs/python3/spkg-configure.m4: Rename config_venv -> conftest_venv
b1518b7Makefile [misc-clean): No need to remove config_venv
179d2e5src/bin/sage-env, src/bin/sage-env-config.in: export/use environment variable PYTHON_FOR_VENV
176ce61build/make/Makefile.in, build/make/deps: Create venv in $SAGE_LOCAL using venv.EnvBuilder, instead of in $SAGE_LOCAL/lib/sage/venv using venv command line
e29087abuild/pkgs/python3/spkg-configure.m4: Only set PYTHON_FOR_VENV if we use the system python
f7924ebbuild/bin/sage-venv: New, use it in build/pkgs/python3/spkg-configure.m4 and build/make/deps

comment:85 Changed 8 months ago by mkoeppe

  • Dependencies #29013 deleted
  • Description modified (diff)
  • Milestone changed from sage-9.2 to sage-9.1

This ticket is no longer on top of #29013.

comment:87 Changed 8 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from new to needs_review

comment:88 Changed 8 months ago by git

  • Commit changed from f7924eb3b6140a725a5181c71936fb9ed65ec5db to c570d6b8b62ff5e040501854329f431309cad6a3

Branch pushed to git repo; I updated commit sha1. New commits:

c570d6bbuild/bin/sage-venv: Fixup

comment:90 Changed 8 months ago by mkoeppe

  • Cc jhpalmieri added

comment:91 Changed 8 months ago by mkoeppe

  • Cc vdelecroix added

comment:93 Changed 8 months ago by mkoeppe

  • Status changed from needs_review to needs_work

Breaks python 2 builds

comment:94 Changed 8 months ago by git

  • Commit changed from c570d6b8b62ff5e040501854329f431309cad6a3 to 65a9f028c2179ff3757f200aae62c8746be6d45c

Branch pushed to git repo; I updated commit sha1. New commits:

65a9f02build/make/deps: Do not set PYTHON = python3_venv in a python2 build

comment:95 Changed 8 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:96 Changed 8 months ago by mkoeppe

Ready for review

comment:98 Changed 8 months ago by git

  • Commit changed from 65a9f028c2179ff3757f200aae62c8746be6d45c to 1c845a4169769157787cdc3e2df37c08a6914891

Branch pushed to git repo; I updated commit sha1. New commits:

1c845a4build/pkgs/python3/spkg-configure.m4: Do not use venv for python3 if we do a python2 build

comment:100 Changed 8 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

Let's move this, looks good to me.

comment:101 Changed 8 months ago by mkoeppe

Thanks!

comment:102 Changed 8 months ago by vbraun

  • Branch changed from public/27824-python3-spkg-configure to 1c845a4169769157787cdc3e2df37c08a6914891
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:103 follow-up: Changed 6 months ago by mmezzarobba

  • Commit 1c845a4169769157787cdc3e2df37c08a6914891 deleted

Hello everyone,

When trying to upgrade to 9.1.rc5, I'm seeing an issue that I suspect could be a side effect of this ticket.

For some reason my sage/local/bin/python3.7 was a symlink to the system python binary. Sage decided it wanted to install its own python-3.7.3.p1, but that failed with a permission error on the symlink. After deleting the symlink, python builds and installs correctly. (I let the logs be overwritten, so I won't be able to provide much details — sorry.)

The overall Sage build still fails with an error about six not being found while building setuptools that I haven't yet investigated.

comment:104 Changed 6 months ago by dimpase

if you switch to py3 from py2, you probably need to run ./configure --with-python=3

it is not clear what version you were on, but the changes to the build system were very substantial, so ymmv.

comment:105 in reply to: ↑ 103 Changed 6 months ago by mkoeppe

Yes, this is a known limitation. I have created #29708 (Fix switching between --with-system-python3 and --without-system-python3) for it. We shall fix it in 9.2.

Note: See TracTickets for help on using tickets.