#27824 closed enhancement (fixed)
spkg-configure.m4 for python3
Reported by: | dimpase | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.1 |
Component: | build: configure | Keywords: | |
Cc: | jdemeyer, embray, vbraun, fbissey, jhpalmieri, vdelecroix | Merged in: | |
Authors: | Matthias Koeppe, Erik Bray | Reviewers: | Dima Pasechnik |
Report Upstream: | N/A | Work issues: | |
Branch: | 1c845a4 (Commits, GitHub, GitLab) | Commit: | |
Dependencies: | Stopgaps: |
Description (last modified by )
Do not install python3 if a sufficiently new version is available in $PATH
. This could be a system python3, or the python3 from within a venv.
As suggested in #29032, we make $SAGE_LOCAL a venv (https://docs.python.org/3/library/venv.html) over the system python3 -- if a suitable system python3 is found -- by using the venv.EnvBuilder
API.
This is done by the new script build/bin/sage-venv
, which we use both during configure
(to make sure that we select a system python3 for which venvs work correctly) and during make
.
The venv does not include the system site-packages: We continue to install all Python packages into our venv using the existing build infrastructure. We keep the task of using system site-packages for the follow-up ticket #29023.
(Note, we use venv
(new since Python 3.3), not virtualenv
. So this change is limited to Python 3 builds of Sage.)
Attachments (1)
Change History (106)
comment:1 Changed 3 years ago by
- Description modified (diff)
comment:2 Changed 3 years ago by
- Description modified (diff)
comment:3 Changed 3 years ago by
- Milestone sage-8.8 deleted
comment:4 Changed 3 years ago by
For Python packages when installing using the system Python we definitely need a well thought out approach, likely with different options for handling questions such as where to install additional Python packages.
$SAGE_LOCAL
already acts as a sort-of virtualenv, but we might still consider using virtualenv/venv in $SAGE_LOCAL
(with the --system-site-packages
option) as a way of extending sys.path
while still using as many system Python packages as possible.
comment:5 Changed 3 years ago by
See #29013 - "In a python3 build, install all Python packages into a venv"
comment:6 Changed 3 years ago by
- Dependencies set to #29013
- Description modified (diff)
- Summary changed from spkg-configure.m4 for pythons to spkg-configure.m4 for python3
Narrowed the ticket to py3.
comment:7 Changed 3 years ago by
- Branch set to public/27824-python3-spkg-configure
comment:8 Changed 3 years ago by
- Commit set to a84815b72ab41f6d0d17b733f44e4e8ec05d3410
Here's a version that works. The venv is created in build/make/Makefile
target (in #29013, I created the venv in the spkg-postinst
script of the python3
spkg.)
New commits:
1363425 | In a python3 build, install all Python packages into a venv
|
a394268 | sage-pip-install: Use PYTHON=sage-python23
|
a84815b | spkg-configure.m4 for python3
|
comment:9 Changed 3 years ago by
- Milestone set to sage-9.2
comment:10 Changed 3 years ago by
how about checking for Py3 deps: sqlite libpng bzip2 xz libffi
?
(I skipped in this list ones that are 2nd order, so no need to check them)
comment:11 Changed 3 years ago by
On Gentoo Python 3 does not have sqlite module by default. I had to do
$ pip install pysqlite3 --user
, and still elliptic_curves
spkg installation errors out.
comment:12 Changed 3 years ago by
- Commit changed from a84815b72ab41f6d0d17b733f44e4e8ec05d3410 to 0d0495fd61f789981d3074ae142b89e248ee394f
Branch pushed to git repo; I updated commit sha1. New commits:
0d0495f | build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi
|
comment:13 Changed 3 years ago by
- Commit changed from 0d0495fd61f789981d3074ae142b89e248ee394f to 98dff9d32325d99c447132a28c4eae6c2d242d9a
Branch pushed to git repo; I updated commit sha1. New commits:
98dff9d | Remove use of --system-site-packages
|
comment:14 follow-up: ↓ 18 Changed 3 years ago by
- Cc jdemeyer embray vbraun added
- Description modified (diff)
I've reduced the scope of the ticket, excluding the use of system site packages.
comment:15 Changed 3 years ago by
- Commit changed from 98dff9d32325d99c447132a28c4eae6c2d242d9a to 61801321f81917550a065462422bb2a209bc1030
Branch pushed to git repo; I updated commit sha1. New commits:
6180132 | build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module
|
comment:16 follow-up: ↓ 24 Changed 3 years ago by
I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with
RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work
even though libSingular
has built just fine (it's some magic with from sage.env import SINGULAR_SO
that is broken, so SINGULAR_SO
is None
)
comment:17 Changed 3 years ago by
Thanks for testing!
comment:18 in reply to: ↑ 14 ; follow-up: ↓ 19 Changed 3 years ago by
Replying to mkoeppe:
I've reduced the scope of the ticket, excluding the use of system site packages.
do you mean pip/etc-installed site-packages
of system's Python?
comment:19 in reply to: ↑ 18 Changed 3 years ago by
comment:20 Changed 3 years ago by
- Dependencies changed from #29013 to #29013, #29019
comment:21 Changed 3 years ago by
- Commit changed from 61801321f81917550a065462422bb2a209bc1030 to 5930b4e219e3f12fb675ba2caf5c1d7fcb1b15e4
Branch pushed to git repo; I updated commit sha1. Last 10 new commits:
61d7822 | Trac #29002: lower the version bound on the system sqlite check.
|
3ab4515 | Merge branch 'u/mjo/ticket/29002' of git://trac.sagemath.org/sage into t/27824/public/27824-python3-spkg-configure
|
c7b9ae4 | Add sqlite depcheck
|
961f850 | Trac #29002: lower the version bound on the system sqlite check (again).
|
d321948 | Trac #29002: drop the readline dependency check for system sqlite.
|
14d15fb | Merge branch 'u/mjo/ticket/29002' of git://trac.sagemath.org/sage into t/27824/public/27824-python3-spkg-configure
|
258d112 | build/pkgs/pillow/patches/setup.py.patch: Don't fail if Py_MACOS_SYSROOT sysconfig variable does not exist
|
43673c0 | Merge branch 't/29019/make_patch_to_pillow_more_robust__do_not_depend_on__py_macos_sysroot__' into t/27824/public/27824-python3-spkg-configure
|
56b1618 | build/pkgs/pillow/spkg-install: Help setup.py find zlib.h on macOS; show more build output
|
5930b4e | Merge branch 't/29019/make_patch_to_pillow_more_robust__do_not_depend_on__py_macos_sysroot__' into t/27824/public/27824-python3-spkg-configure
|
comment:22 Changed 3 years ago by
Now this is giving
[pillow-5.3.0.p0] gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -g -DHAVE_OPENJPEG -DHAVE_LIBZ -DHAVE_LIBTIFF -DPILLOW_VERSION="5.3.0" -I/usr/local/Cellar/openjpeg/2.3.1/include/openjpeg-2.3 -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/var/tmp/sage/build/pillow-5.3.0.p0/src/src/libImaging -I/usr/local/Cellar/jpeg/9c/include -I/usr/local/Cellar/libtiff/4.1.0/include -I/usr/local/Cellar/freetype/2.10.1/include/freetype2 -I/usr/local/Cellar/little-cms2/2.9/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/lib/sage/venv/sage/include -I/usr/local/include -I/usr/local/Cellar/freetype/2.10.1/include -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/mkoeppe/s/sage/sage-rebasing/worktree-venv/local/lib/sage/venv/sage/include -I/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/include/python3.7m -c src/_imaging.c -o build/temp.macosx-10.15-x86_64-3.7/src/_imaging.o [pillow-5.3.0.p0] error: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.9" but "10.15" during configure [pillow-5.3.0.p0] ********************************************************************************
This is coming from the fake MACOSX_DEPLOYMENT_TARGET
set in sage-env
. See #18272, #7095, #18254, #16312.
comment:23 Changed 3 years ago by
- Commit changed from 5930b4e219e3f12fb675ba2caf5c1d7fcb1b15e4 to 7bd87464eb40adfe7f9110a452cd10b3c78b8f4c
comment:24 in reply to: ↑ 16 ; follow-up: ↓ 29 Changed 3 years ago by
Replying to dimpase:
I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with
RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to workeven though
libSingular
has built just fine (it's some magic withfrom sage.env import SINGULAR_SO
that is broken, soSINGULAR_SO
isNone
)
Fixed now.
comment:25 Changed 3 years ago by
- Description modified (diff)
comment:26 Changed 3 years ago by
- Commit changed from 7bd87464eb40adfe7f9110a452cd10b3c78b8f4c to cd864e89500858a87f0f82faf4a634d200fe2118
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
d4fbe40 | _get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
|
bba08ec | Create module src/sage/env_config.py from src/sage/env_config.py.in, defining SAGE_LOCAL
|
d1779cc | Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
|
cd864e8 | Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure
|
comment:27 Changed 3 years ago by
Compiles OK on macOS with homebrew. make ptestlong
mostly good except for
- GLPK output (spkg-configure should really reject a libglpk that talks too much) - it makes it hard to see actual failures
- and numerical failures in
src/sage/matrix/matrix2.pyx
,src/sage/finance/time_series.pyx
, andsrc/sage/schemes/affine/affine_homset.py
as reported in #29013sage -t --long src/sage/matrix/matrix2.pyx # 2 doctests failed sage -t --long src/sage/finance/time_series.pyx # 5 doctests failed sage -t --long src/sage/schemes/affine/affine_homset.py # 1 doctest failed
comment:28 Changed 3 years ago by
- Commit changed from cd864e89500858a87f0f82faf4a634d200fe2118 to 669b1dfa8bfd50c0710f546ed2bdfe43b14bc13d
Branch pushed to git repo; I updated commit sha1. New commits:
85147f9 | build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg
|
93bae6f | Merge branch 't/29025/numpy__site_cfg_needs_a__default__section' into t/29013/public/29013-use-py3-venv
|
669b1df | Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure
|
comment:29 in reply to: ↑ 24 Changed 3 years ago by
Replying to mkoeppe:
Replying to dimpase:
I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python) and it broke at docbuild, with
RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to workeven though
libSingular
has built just fine (it's some magic withfrom sage.env import SINGULAR_SO
that is broken, soSINGULAR_SO
isNone
)Fixed now.
after this fix, it builds and passes most tests on Gentoo (some errors related to cryptominisat that I had installed, and it probably got broken in some way)
comment:30 Changed 3 years ago by
I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)
sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py ********************************************************************** File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat Failed example: solver = CryptoMiniSat() # optional - cryptominisat Exception raised: Traceback (most recent call last): File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__ from pycryptosat import Solver ModuleNotFoundError: No module named 'pycryptosat'
I wonder whether it uses a convoluted way to create python extensions, which got broken.
comment:31 Changed 3 years ago by
I see (on a system where cryptominisat package works) that it is one of few packages that get installed without a sub-directory named the same as package in site-packages:
sage: import pycryptosat sage: print(pycryptosat.__file__) /home/scratch2/dimpase/sage/sage/local/lib/python3.7/site-packages/pycryptosat.cpython-37m-x86_64-linux-gnu.so sage: import numpy sage: print(numpy.__file__) /home/scratch2/dimpase/sage/sage/local/lib/python3.7/site-packages/numpy/__init__.py
Not sure whether this is a bug on the package, or a bug in this branch.
On this branch I see (with local=SAGE_LOCAL
)
$ ls local/lib/python3.7/site-packages/* local/lib/python3.7/site-packages/pycryptosat-0.2.0-py3.7.egg-info local/lib/python3.7/site-packages/pycryptosat.cpython-37m-x86_64-linux-gnu.so
So it seems that this branch does not pick up anything from that location.
comment:32 Changed 3 years ago by
- Commit changed from 669b1dfa8bfd50c0710f546ed2bdfe43b14bc13d to 61a9e3733989f265bda6cea7e6bccf7bf88c9749
comment:33 Changed 3 years ago by
- Commit changed from 61a9e3733989f265bda6cea7e6bccf7bf88c9749 to 8c546cfd4a35c82c911e3620538c2188e7e350f2
comment:34 follow-up: ↓ 36 Changed 3 years ago by
What's the deal with config_venv
? Why is any of that necessary? I looked at the commit that added it but it wasn't clear. It seems to me another reason to avoid all the venv stuff...
comment:35 follow-up: ↓ 37 Changed 3 years ago by
So, all that $VIRTUAL_ENV/bin/activate
does, essentially, is to set 3 environment variables: PATH
(to which it inserts $VIRTUAL_ENV/bin
in the front), PS1
(which Sage also already sets when activating the sage environment), and the $VIRTUAL_ENV
environment variable itself.
Other than that, the only thing that really makes a directory a virtualenv (with Python 3 + venv, I'm not counting the old virtualenv) is the presence of pyvenv.cfg
in the install prefix. The format of this file is more-or-less documented (albeit not very clearly) in https://docs.python.org/3/library/venv.html. So all we really need to do to turn $SAGE_LOCAL
into a virtualenv of a system Python 3 is to write the appropriate pyvenv.cfg
into $SAGE_LOCAL
, and not mess around with the full venv
command at all. That, and make $SAGE_LOCAL/bin/python3
a symlink to the real python3.
Although not strictly necessary, it is also possibly helpful to third-party tools to set the $VIRTUAL_ENV
environment variable.
comment:36 in reply to: ↑ 34 ; follow-up: ↓ 41 Changed 3 years ago by
Replying to embray:
What's the deal with
config_venv
? Why is any of that necessary? I looked at the commit that added it but it wasn't clear.
It is used in a configure test, build/pkgs/python3/spkg-configure.m4
to make sure that the system python has all optional modules that we need.
comment:37 in reply to: ↑ 35 Changed 3 years ago by
Replying to embray:
So all we really need to do to turn
$SAGE_LOCAL
into a virtualenv of a system Python 3 is to write the appropriatepyvenv.cfg
into$SAGE_LOCAL
, and not mess around with the fullvenv
command at all.
This amounts to trying to upgrade python3's venv
to something like a conda environment. This is what I avoid in #29013 on purpose - the venv
is only used in a clean, documented way, using the documented tools.
comment:38 follow-ups: ↓ 39 ↓ 40 Changed 3 years ago by
writing pyvenv.cfg
by hand is bound to be broken at some point, as it's an implementation detail of venv.
The correct interface to venv
is to use venv
, I think.
(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))
comment:39 in reply to: ↑ 38 ; follow-up: ↓ 44 Changed 3 years ago by
Replying to dimpase:
(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))
Glad we have this on record now for the software historians/archaeologists to discover :)
comment:40 in reply to: ↑ 38 Changed 3 years ago by
Replying to dimpase:
writing
pyvenv.cfg
by hand is bound to be broken at some point, as it's an implementation detail of venv. The correct interface tovenv
is to usevenv
, I think.
That's not true. It's documented both in the documentation for the venv
module and the site
module. If you really want to avoid writing it by hand the venv
module includes code for that as well. This (with the system Python 3) is effectively the same (albeit more opaque). From within SAGE_ROOT, but without the sage environment activated:
$ python3 -c 'import venv; b = venv.EnvBuilder(); c = b.ensure_directories("local"); b.create_configuration(c)' $ cat local/pyvenv.cfg home = /usr/bin include-system-site-packages = false version = 3.6.8
but I think even that's overkill for now.
comment:41 in reply to: ↑ 36 ; follow-up: ↓ 42 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
What's the deal with
config_venv
? Why is any of that necessary? I looked at the commit that added it but it wasn't clear.It is used in a configure test,
build/pkgs/python3/spkg-configure.m4
to make sure that the system python has all optional modules that we need.
But why do you need to make a virtualenv for that? Why not run the system python directly?
comment:42 in reply to: ↑ 41 ; follow-up: ↓ 47 Changed 3 years ago by
Replying to embray:
But why do you need to make a virtualenv for that? Why not run the system python directly?
On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.
comment:43 Changed 3 years ago by
I'll add this documentation to the branch.
comment:44 in reply to: ↑ 39 Changed 3 years ago by
Replying to mkoeppe:
Replying to dimpase:
(By the way, the idea of using venv/virtualenv is probably mine, IIRC I put it in the original version of this ticket :-))
Glad we have this on record now for the software historians/archaeologists to discover :)
it's here: https://trac.sagemath.org/ticket/27824?version=0
comment:45 Changed 3 years ago by
- Commit changed from 8c546cfd4a35c82c911e3620538c2188e7e350f2 to 348cf2243ce7c8696504a5d85eee849c24cef2a9
Branch pushed to git repo; I updated commit sha1. New commits:
348cf22 | build/pkgs/python3/spkg-configure.m4: Add comments
|
comment:46 follow-up: ↓ 50 Changed 3 years ago by
By the way, should I be checking for the ssl module too?
comment:47 in reply to: ↑ 42 ; follow-up: ↓ 48 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
But why do you need to make a virtualenv for that? Why not run the system python directly?
On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.
But why?
comment:48 in reply to: ↑ 47 ; follow-up: ↓ 49 Changed 3 years ago by
Replying to embray:
Replying to mkoeppe:
Replying to embray:
But why do you need to make a virtualenv for that? Why not run the system python directly?
On this ticket, the system python is not used directly but rather in a venv without site-packages. Therefore the correct test is to test whether the module will be available in a venv.
But why?
Wait, I misunderstood what you wrote at first. I know why it uses a virtualenv. What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?
comment:49 in reply to: ↑ 48 ; follow-ups: ↓ 51 ↓ 63 Changed 3 years ago by
Replying to embray:
What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?
Because system python could have a module named sqlite3
in its site-packages
; but then when we try to use it in a venv, it won't be there.
comment:50 in reply to: ↑ 46 Changed 3 years ago by
comment:51 in reply to: ↑ 49 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?
Because system python could have a module named
sqlite3
in itssite-packages
; but then when we try to use it in a venv, it won't be there.
So use python3 -S
if you really want to be sure. That disables site-packages.
comment:52 follow-up: ↓ 55 Changed 3 years ago by
An interesting discovery while playing around with this: By default Ubuntu's Python does not have distutils. You have to explicitly apt-get install python3-distutils
. So this is something we also need to check for at configure time or else it wont be possible to install Python packages.
comment:53 follow-up: ↓ 69 Changed 3 years ago by
FWIW I'm experimenting with an alternative solution based on my comments here (albeit with significant pieces taken from this ticket!) I'll update when I have something that I think works...
comment:54 Changed 3 years ago by
- Commit changed from 348cf2243ce7c8696504a5d85eee849c24cef2a9 to cb399a1fcfd4d8a74c90bad95d1e5d7e521f19a1
Branch pushed to git repo; I updated commit sha1. New commits:
cb399a1 | Set PYTHON_FOR_VENV correctly, use it in build/make/deps
|
comment:55 in reply to: ↑ 52 Changed 3 years ago by
Replying to embray:
By default Ubuntu's Python does not have distutils. You have to explicitly
apt-get install python3-distutils
. So this is something we also need to check for at configure time or else it wont be possible to install Python packages.
Thanks! I'll add this test.
comment:56 Changed 3 years ago by
- Commit changed from cb399a1fcfd4d8a74c90bad95d1e5d7e521f19a1 to 71cc4919119784d243d65f853f336771e93e4478
Branch pushed to git repo; I updated commit sha1. New commits:
71cc491 | build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
|
comment:57 Changed 3 years ago by
Also, of course, one needs the Python headers, on Ubuntu from libpython3.7-dev
.
comment:58 Changed 3 years ago by
So I'll add a check that the distutils can actually build an extension.
comment:59 Changed 3 years ago by
- Commit changed from 71cc4919119784d243d65f853f336771e93e4478 to c7c16711c2f3ab262c8d76f6b565727bfc01edff
Branch pushed to git repo; I updated commit sha1. New commits:
c7c1671 | build/pkgs/python3/spkg-configure.m4: Check that distutils can build an extension
|
Changed 3 years ago by
comment:60 follow-up: ↓ 61 Changed 3 years ago by
Testing with the attached Dockerfile on ubuntu bionic.
module and distutils checking seems to work correctly.
Got stuck at scipy
install.
[scipy-1.2.0] numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.
Looks like scipy
would need a sitecfg as well so that it finds BLAS. It's strange that we build numpy with providing site.cfg
(which we just had to correct in #29025), but scipy
without site.cfg
.
comment:61 in reply to: ↑ 60 Changed 3 years ago by
- Cc fbissey added
Replying to mkoeppe:
Testing with the attached Dockerfile on ubuntu bionic.
module and distutils checking seems to work correctly.
Got stuck at
scipy
install.[scipy-1.2.0] numpy.distutils.system_info.NotFoundError: No lapack/blas resources found.Looks like
scipy
would need a sitecfg as well so that it finds BLAS. It's strange that we build numpy with providingsite.cfg
(which we just had to correct in #29025), butscipy
withoutsite.cfg
.
@fbissey, based on your work on scipy configuration in #20157, would you be able to help with this?
comment:62 follow-up: ↓ 68 Changed 3 years ago by
You could provide an identical site.cfg
for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a file numpy/distutils/site.cfg
in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.
comment:63 in reply to: ↑ 49 ; follow-up: ↓ 64 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
What I don't get is why you need to make a venv to test features of the system python when you can just call the system python directly here?
Because system python could have a module named
sqlite3
in itssite-packages
; but then when we try to use it in a venv, it won't be there.
I wonder about rationale for disabling system's site-packages
.
Is this the only way to install custom versions of packages in venv (which can conflict with ones is system's site-packages
)?
comment:64 in reply to: ↑ 63 ; follow-up: ↓ 65 Changed 3 years ago by
I wonder about rationale for disabling system's
site-packages
. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system'ssite-packages
)?
It’s strictly for making this ticket easier because I don’t have to worry about getting the correct versions of all Python packages. See #29023 for the next step - let’s discuss any issues regarding use of system site packages there.
comment:65 in reply to: ↑ 64 Changed 3 years ago by
Replying to mkoeppe:
I wonder about rationale for disabling system's
site-packages
. Is this the only way to install custom versions of packages in venv (which can conflict with ones is system'ssite-packages
)?It’s strictly for making this ticket easier because I don’t have to worry about getting the correct versions of all Python packages. See #29023 for the next step - let’s discuss any issues regarding use of system site packages there.
I agree here, it's much simpler for now to focus on using a "bare" Python as much as possible. I think detecting and using system Python packages is where we want to go, but it can get a bit hairier...
comment:66 Changed 3 years ago by
For the heck of it, I'm seeing how far I can get with a system Python 3.6. I think anything less than 3.6 is almost definitely too old, but I have a feeling 3.6 might still be usable as a minimum. The only major hassle I've had so far is ensuring that it uses UTF-8 encoding by default.
comment:67 Changed 3 years ago by
- Dependencies changed from #29013, #29019 to #29002, #29013, #29019
comment:68 in reply to: ↑ 62 ; follow-up: ↓ 75 Changed 3 years ago by
Replying to fbissey:
You could provide an identical
site.cfg
for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a filenumpy/distutils/site.cfg
in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.
Thank you! The file is installed. I'll investigate why it is not enough.
comment:69 in reply to: ↑ 53 Changed 3 years ago by
Replying to embray:
FWIW I'm experimenting with an alternative solution based on my comments here (albeit with significant pieces taken from this ticket!) I'll update when I have something that I think works...
Here's a prototype of my alternative approach: #29032
I admit it does not yet address as many issues as this ticket does, like all the site.cfg fixes with Numpy. Personally I didn't need it, but if those fixes are necessary they can easily be applied to my ticket as well.
comment:70 Changed 3 years ago by
- Commit changed from c7c16711c2f3ab262c8d76f6b565727bfc01edff to 356b6d78d66202a17ebcace6178baea91373deb3
Branch pushed to git repo; I updated commit sha1. New commits:
356b6d7 | build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
|
comment:71 Changed 3 years ago by
(extracted from Erik's commit at #29032.)
comment:72 Changed 3 years ago by
- Commit changed from 356b6d78d66202a17ebcace6178baea91373deb3 to 92953ceda3f7ba7d6b0246fa5828372bf352dc84
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
92953ce | build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
|
comment:73 Changed 3 years ago by
comment:74 Changed 3 years ago by
- Commit changed from 92953ceda3f7ba7d6b0246fa5828372bf352dc84 to 4c78bdfccf72183181ee666909dbe365f50b7143
Branch pushed to git repo; I updated commit sha1. New commits:
08ec551 | build/make/deps: Make python3_venv a .PHONY target
|
b2ab6f3 | src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
|
7a43544 | src/sage/env.py, envconfig.py.in: Add MAXIMA; src/sage/interfaces/maxima.py: Use it
|
f2af478 | Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
|
8959c19 | Merge branch 't/29013/public/29013-use-py3-venv' into t/27824/public/27824-python3-spkg-configure
|
4c78bdf | src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032
|
comment:75 in reply to: ↑ 68 Changed 3 years ago by
Replying to mkoeppe:
Replying to fbissey:
You could provide an identical
site.cfg
for scipy. The reason you shouldn't technically have to do it, is that the one from numpy is actually shipped. You should have a filenumpy/distutils/site.cfg
in your numpy install and it should be the one you used to configure numpy. If it is missing you'll run into the trouble above I am sure.Thank you! The file is installed. I'll investigate why it is not enough.
I've created ticket #29051 for this (Paths configured in installed numpy site.cfg [DEFAULT/ALL]
do not affect scipy)
comment:76 Changed 2 years ago by
- Commit changed from 4c78bdfccf72183181ee666909dbe365f50b7143 to 3d90535b3301e49433e98146e2179c6daa549bf2
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
56e3daa | src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
|
208c7e6 | src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL
|
1bd8aef | spkg-configure.m4 for python3
|
7237aa4 | build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi
|
d044744 | build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module
|
c6fec13 | src/bin/sage-env: Don't set MACOSX_DEPLOYMENT_TARGET if we use system python3
|
b69ac5b | Set PYTHON_FOR_VENV correctly, use it in build/make/deps
|
c143043 | build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
|
973c36f | build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
|
3d90535 | src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032
|
comment:77 Changed 2 years ago by
- Dependencies changed from #29002, #29013, #29019 to #29013
comment:78 Changed 2 years ago by
- Commit changed from 3d90535b3301e49433e98146e2179c6daa549bf2 to b7c5335886a174712d11b3d24a7b485e4d30a820
Branch pushed to git repo; I updated commit sha1. New commits:
73e730c | build/bin/sage-system-python: Prefer python3, check a larger list of executable names, check for urllib; exit with a proper error message if no suitable Python is found
|
b7c5335 | configure.ac: Stop early if no suitable Python is found
|
comment:79 Changed 2 years ago by
- Commit changed from b7c5335886a174712d11b3d24a7b485e4d30a820 to 3aa209ef8ca3605b78dbdd7fa59b3a337593ac6e
Branch pushed to git repo; I updated commit sha1. New commits:
3aa209e | src/bin/sage-location: Use sage-python instead of sage-system-python
|
comment:80 Changed 2 years ago by
- Commit changed from 3aa209ef8ca3605b78dbdd7fa59b3a337593ac6e to 3d90535b3301e49433e98146e2179c6daa549bf2
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
comment:81 Changed 2 years ago by
(pushed to wrong ticket, corrected)
comment:82 Changed 2 years ago by
- Commit changed from 3d90535b3301e49433e98146e2179c6daa549bf2 to dfa28169a5a2c20846df1f2f134e513a66091ede
comment:83 Changed 2 years ago by
- Commit changed from dfa28169a5a2c20846df1f2f134e513a66091ede to 8f441234e408c4a45e3ebe329d8e96c12e0da78e
Branch pushed to git repo; I updated commit sha1. New commits:
8f44123 | src/bin/sage-env, src/bin/sage-env-config.in: export/use environment variable PYTHON_FOR_VENV
|
comment:84 Changed 2 years ago by
- Commit changed from 8f441234e408c4a45e3ebe329d8e96c12e0da78e to f7924eb3b6140a725a5181c71936fb9ed65ec5db
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
c0a99e5 | Set PYTHON_FOR_VENV correctly, use it in build/make/deps
|
6fcd7cd | build/pkgs/python3/spkg-configure.m4: Check for more modules, including distutils.core
|
15eceae | build/pkgs/python3/spkg-configure.m4: Rename config_setup.py and config_build to conftest.py and conftest.dir respectively since autoconf will automatically clean up files and directories beginning with "conftest"
|
fde251b | src/bin/sage-env: Make inst_python3_venv = $(SAGE_VENV)/pyvenv.cfg. Adapted from #29032
|
e73a428 | build/pkgs/python3/spkg-configure.m4: Rename config_venv -> conftest_venv
|
b1518b7 | Makefile [misc-clean): No need to remove config_venv
|
179d2e5 | src/bin/sage-env, src/bin/sage-env-config.in: export/use environment variable PYTHON_FOR_VENV
|
176ce61 | build/make/Makefile.in, build/make/deps: Create venv in $SAGE_LOCAL using venv.EnvBuilder, instead of in $SAGE_LOCAL/lib/sage/venv using venv command line
|
e29087a | build/pkgs/python3/spkg-configure.m4: Only set PYTHON_FOR_VENV if we use the system python
|
f7924eb | build/bin/sage-venv: New, use it in build/pkgs/python3/spkg-configure.m4 and build/make/deps
|
comment:85 Changed 2 years ago by
- Dependencies #29013 deleted
- Description modified (diff)
- Milestone changed from sage-9.2 to sage-9.1
This ticket is no longer on top of #29013.
comment:86 Changed 2 years ago by
comment:87 Changed 2 years ago by
- Description modified (diff)
- Status changed from new to needs_review
comment:88 Changed 2 years ago by
- Commit changed from f7924eb3b6140a725a5181c71936fb9ed65ec5db to c570d6b8b62ff5e040501854329f431309cad6a3
Branch pushed to git repo; I updated commit sha1. New commits:
c570d6b | build/bin/sage-venv: Fixup
|
comment:89 Changed 2 years ago by
A successful test at: https://github.com/mkoeppe/sage/runs/520449412
comment:90 Changed 2 years ago by
- Cc jhpalmieri added
comment:91 Changed 2 years ago by
- Cc vdelecroix added
comment:92 Changed 2 years ago by
comment:93 Changed 2 years ago by
- Status changed from needs_review to needs_work
Breaks python 2 builds
comment:94 Changed 2 years ago by
- Commit changed from c570d6b8b62ff5e040501854329f431309cad6a3 to 65a9f028c2179ff3757f200aae62c8746be6d45c
Branch pushed to git repo; I updated commit sha1. New commits:
65a9f02 | build/make/deps: Do not set PYTHON = python3_venv in a python2 build
|
comment:95 Changed 2 years ago by
- Status changed from needs_work to needs_review
comment:96 Changed 2 years ago by
Ready for review
comment:97 Changed 2 years ago by
New tests at https://github.com/mkoeppe/sage/actions/runs/60877259; see also #29367
comment:98 Changed 2 years ago by
- Commit changed from 65a9f028c2179ff3757f200aae62c8746be6d45c to 1c845a4169769157787cdc3e2df37c08a6914891
Branch pushed to git repo; I updated commit sha1. New commits:
1c845a4 | build/pkgs/python3/spkg-configure.m4: Do not use venv for python3 if we do a python2 build
|
comment:99 Changed 2 years ago by
comment:100 Changed 2 years ago by
- Reviewers set to Dima Pasechnik
- Status changed from needs_review to positive_review
Let's move this, looks good to me.
comment:101 Changed 2 years ago by
Thanks!
comment:102 Changed 2 years ago by
- Branch changed from public/27824-python3-spkg-configure to 1c845a4169769157787cdc3e2df37c08a6914891
- Resolution set to fixed
- Status changed from positive_review to closed
comment:103 follow-up: ↓ 105 Changed 2 years ago by
- Commit 1c845a4169769157787cdc3e2df37c08a6914891 deleted
Hello everyone,
When trying to upgrade to 9.1.rc5, I'm seeing an issue that I suspect could be a side effect of this ticket.
For some reason my sage/local/bin/python3.7
was a symlink to the system python binary. Sage decided it wanted to install its own python-3.7.3.p1, but that failed with a permission error on the symlink. After deleting the symlink, python builds and installs correctly. (I let the logs be overwritten, so I won't be able to provide much details — sorry.)
The overall Sage build still fails with an error about six not being found while building setuptools that I haven't yet investigated.
comment:104 Changed 2 years ago by
if you switch to py3 from py2, you probably need to run ./configure --with-python=3
it is not clear what version you were on, but the changes to the build system were very substantial, so ymmv.
comment:105 in reply to: ↑ 103 Changed 2 years ago by
Yes, this is a known limitation. I have created #29708 (Fix switching between --with-system-python3 and --without-system-python3) for it. We shall fix it in 9.2.
As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).