Opened 3 years ago
Closed 15 months ago
#29013 closed enhancement (fixed)
Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL
Reported by: | mkoeppe | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-9.4 |
Component: | build | Keywords: | sd111 |
Cc: | dimpase, jdemeyer, embray, vbraun, jhpalmieri | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | John Palmieri |
Report Upstream: | N/A | Work issues: | |
Branch: | d68e861 (Commits, GitHub, GitLab) | Commit: | d68e86144b4a37600654899d2ab655ae55fb4378 |
Dependencies: | Stopgaps: |
Description (last modified by )
#27824 made $SAGE_LOCAL
a venv if a suitable system python3 is found.
In this ticket, we add support for installing builds of Sage with several python versions, sharing the non-Python packages in SAGE_LOCAL
:
- keeping install records for Python packages (recognized by having
install-requires.txt
orrequirements.txt
) in$SAGE_VENV/var/lib/sage/{installed,scripts,wheels}
, separate from those of non-Python packages ($SAGE_LOCAL/var/lib/sage/{installed,scripts}
) - likewise for
$SAGE_LOCAL/var/tmp/sage/build
- here
$SAGE_VENV
defaults to$SAGE_LOCAL
, but can be overridden to an arbitrary directory that will be used as the wheel-building venv, for example$SAGE_LOCAL/var/lib/sage/venv/$PYTHON_TAG
.
Configuration:
- This is activated by running, for example,
./configure --with-python=/usr/bin/python3.8 --with-sage-venv="$SAGE_LOCAL/var/lib/sage/venv/python3.8"
.
As of this ticket, we are able to establish the venv in an arbitrary configured directory instead of SAGE_LOCAL
. This is already useful for #31396 to create a wheel-building venv that will not be packaged as part of SAGE_LOCAL
.
In follow-up tickets, we gain the full functionality that facilitates testing with several Python versions without having to rebuild the Sage distribution. This depends on:
- #30534 Repackage
pynac
as a pip-installable package
Follow-ups:
- We also want to support
./configure --with-sage-venv=no
which would suppress making any venv (#30896)
- Support the tricky case: When system python3 is not in use and SAGE_VENV != SAGE_LOCAL, then both a real
$SAGE_LOCAL/bin/python3
needs to be built and a venv in SAGE_VENV created... or should python3 be installed in$SAGE_VENV
?)
- Optional: A file
build/pkgs/SPKG/trees
could override the install tree determination - this would allow us to for example install a package such asjupyter_core
both into something likeSAGE_NOTEBOOK_VENV
andSAGE_VENV
...
Change History (108)
comment:1 Changed 3 years ago by
- Description modified (diff)
comment:2 Changed 3 years ago by
- Branch set to public/29013-use-py3-venv
comment:3 Changed 3 years ago by
- Cc vbraun added
- Commit set to a3942688c3398af5c7bcc62f3fb3dde191a9b2c9
- Status changed from new to needs_review
comment:4 Changed 3 years ago by
In the branch #27824, the venv is created by slightly more complicated code in build/make/Makefile
. This complexity is not necessary for the present ticket.
comment:5 Changed 3 years ago by
Does this ticket make sense separately, or only as part of #27824 ?
comment:6 Changed 3 years ago by
#27824 is the use case that I have in mind. This ticket is a less intrusive change and should go through the betas first.
comment:7 Changed 3 years ago by
- Dependencies set to #29002
comment:8 follow-up: ↓ 10 Changed 3 years ago by
I tried building with this branch on OS X, and I got doctest failures:
sage -t --long src/sage/plot/plot.py # 1 doctest failed sage -t --long src/sage/matrix/matrix2.pyx # 2 doctests failed sage -t --long src/sage/finance/time_series.pyx # 5 doctests failed sage -t --long src/sage/misc/trace.py # 4 doctests failed sage -t --long src/sage/env.py # 1 doctest failed sage -t --long src/sage/repl/configuration.py # 1 doctest failed sage -t --long src/sage/schemes/affine/affine_homset.py # 1 doctest failed
For example
File "src/sage/plot/plot.py", line 521, in sage.plot.plot Failed example: os.system("sage -c \"if 'matplotlib' in sys.modules: sys.exit(1)\"") # long time Expected: 0 Got: Traceback (most recent call last): File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.1.beta0/local/bin/sage-eval", line 4, in <module> from sage.all import * ModuleNotFoundError: No module named 'sage' 256
and
********************************************************************** File "src/sage/matrix/matrix2.pyx", line 1127, in sage.matrix.matrix2.Matrix.pseudoinverse Failed example: M.pseudoinverse() # tol 1e-15 Expected: [0.0620518477661335 0.0206839492553778 0.0124103695532267] [ 0.124103695532267 0.0413678985107557 0.0248207391064534] [ 0.186155543298400 0.0620518477661335 0.0372311086596801] Got: [0.0620518477661334 0.0206839492553778 0.0124103695532267] [ 0.124103695532267 0.0413678985107556 0.0248207391064534] [ 0.186155543298400 0.0620518477661335 0.0372311086596801] Tolerance exceeded in 2 of 9: 0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15 0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15 ********************************************************************** File "src/sage/matrix/matrix2.pyx", line 1131, in sage.matrix.matrix2.Matrix.pseudoinverse Failed example: M.pseudoinverse(algorithm="numpy") # tol 1e-15 Expected: [0.0620518477661335 0.0206839492553778 0.0124103695532267] [ 0.124103695532267 0.0413678985107557 0.0248207391064534] [ 0.186155543298400 0.0620518477661335 0.0372311086596801] Got: [0.0620518477661334 0.0206839492553778 0.0124103695532267] [ 0.124103695532267 0.0413678985107556 0.0248207391064534] [ 0.186155543298400 0.0620518477661335 0.0372311086596801] Tolerance exceeded in 2 of 9: 0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15 0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15 ********************************************************************** 1 item had failures: 2 of 25 in sage.matrix.matrix2.Matrix.pseudoinverse [2472 tests, 2 failures, 9.06 s]
comment:9 Changed 3 years ago by
Thanks for testing!
comment:10 in reply to: ↑ 8 ; follow-up: ↓ 13 Changed 3 years ago by
- Dependencies changed from #29002 to #29002, #29022
Replying to jhpalmieri:
sage -t --long src/sage/matrix/matrix2.pyx # 2 doctests failed
Not sure what's causing the numerical failures.
sage -t --long src/sage/env.py # 1 doctest failed
For this one, I have created #29022.
comment:11 Changed 3 years ago by
- Commit changed from a3942688c3398af5c7bcc62f3fb3dde191a9b2c9 to d4fbe4082890ac0aa09559b87393544678592c94
Branch pushed to git repo; I updated commit sha1. New commits:
d4fbe40 | _get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
|
comment:12 Changed 3 years ago by
- Commit changed from d4fbe4082890ac0aa09559b87393544678592c94 to d1779cc2ff56d8dd5f58ebd1f5322e32499a3970
Branch pushed to git repo; I updated commit sha1. New commits:
bba08ec | Create module src/sage/env_config.py from src/sage/env_config.py.in, defining SAGE_LOCAL
|
d1779cc | Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
|
comment:13 in reply to: ↑ 10 Changed 3 years ago by
comment:14 Changed 3 years ago by
The numerical problems seems to happen because numpy does not find BLAS.
comment:15 Changed 3 years ago by
Actually, it does not find openblas or atlas, but it does find the "Accelerate framework":
accelerate_info: FOUND: extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)] FOUND: extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]
comment:16 Changed 3 years ago by
- Commit changed from d1779cc2ff56d8dd5f58ebd1f5322e32499a3970 to 93bae6f990fa685b12f26a2b2736d45f7c605ce6
comment:17 Changed 3 years ago by
- Dependencies changed from #29002, #29022 to #29002, #29022, #29025
Merged #29025, fixes the numerical doctests.
comment:18 follow-up: ↓ 22 Changed 3 years ago by
I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)
sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py ********************************************************************** File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat Failed example: solver = CryptoMiniSat() # optional - cryptominisat Exception raised: Traceback (most recent call last): File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__ from pycryptosat import Solver ModuleNotFoundError: No module named 'pycryptosat'
I wonder whether it uses a convoluted way to create python extensions, which got broken.
comment:19 Changed 3 years ago by
oops, comment:18 belongs to #27824
comment:20 follow-up: ↓ 21 Changed 3 years ago by
Minor bikeshed, but why the long path "lib/sage/venv/sage"? What about something like just "lib/sage-X.Y"; analogously to "lib/python-X.Y"? We've never had a "lib/sage" before in the first place, so I don't see any immediate need to make the path so much more complicated.
comment:21 in reply to: ↑ 20 ; follow-up: ↓ 44 Changed 3 years ago by
Replying to embray:
Minor bikeshed, but why the long path "lib/sage/venv/sage"? What about something like just "lib/sage-X.Y"; analogously to "lib/python-X.Y"? We've never had a "lib/sage" before in the first place, so I don't see any immediate need to make the path so much more complicated.
1) "venv" has to be in the path to indicate that it's a venv in the technical sense and not a random place with invitation to install stuff in it.
2) Definitely no versioning.
3) I suppose I could drop the final "sage", making it lib/sage/venv
comment:22 in reply to: ↑ 18 Changed 3 years ago by
Replying to dimpase:
I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)
sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py ********************************************************************** File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat Failed example: solver = CryptoMiniSat() # optional - cryptominisat Exception raised: Traceback (most recent call last): File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__ from pycryptosat import Solver ModuleNotFoundError: No module named 'pycryptosat'I wonder whether it uses a convoluted way to create python extensions, which got broken.
Upstream merged pull request https://github.com/msoos/cryptominisat/pull/551
to take care of a virtualenv
installation.
comment:23 Changed 3 years ago by
I haven't tested whether this patch helps for our venv
. The patch is not in the current release 5.6.8 (which we have).
comment:24 Changed 3 years ago by
- Commit changed from 93bae6f990fa685b12f26a2b2736d45f7c605ce6 to 2ed5d071cf9825df2a2dbf6d54b3a6f09153be3a
Branch pushed to git repo; I updated commit sha1. New commits:
2ed5d07 | Add cryptominisat patch for virtual envs
|
comment:25 Changed 3 years ago by
Try with this patch please. I can't compile cryptominisat because of an unrelated reason.
comment:26 Changed 3 years ago by
patch landed into a wrong place, it should be one level down, in patches/
testing now, after correction of this.
comment:27 Changed 3 years ago by
Sorry, thanks.
comment:28 Changed 3 years ago by
- Commit changed from 2ed5d071cf9825df2a2dbf6d54b3a6f09153be3a to 27d945a8d41354234c123f02e2e1d41d287a7541
Branch pushed to git repo; I updated commit sha1. New commits:
27d945a | moved patch to the right place
|
comment:29 Changed 3 years ago by
the patch worked, it's all good with tests now.
comment:30 Changed 3 years ago by
Thank you!
comment:31 follow-up: ↓ 32 Changed 3 years ago by
I might have missed some discussion elsewhere, but I still don't see the advantage of this extra complication over having the sage environment just extend sys.path
where necessary. I say this as a big fan of virtualenvs in general, and having suggested doing something like this in the past. But the more I think about it the less I see any advantage in this case over having an additional sys.path
entry.
comment:32 in reply to: ↑ 31 Changed 3 years ago by
Replying to embray:
I say this as a big fan of virtualenvs in general, and having suggested doing something like this in the past.
Yes, of course this ticket was inspired by various insights and discussions shared by you and others on trac tickets.
But the more I think about it the less I see any advantage in this case over having an additional
sys.path
entry.
Previously, with our python2 build, there was no clear way to do it. But now with python 3 there is a standard and clean tool: that's venv. Installing into it is supported by standard tools.
comment:33 follow-ups: ↓ 34 ↓ 35 Changed 3 years ago by
Or, at the very least, what I've suggested in the past was to make $SAGE_LOCAL
itself a virtualenv, as virtualenvs already have a sort-of root filesystem, including among other things lib/pythonX.Y/site-packages
.
With this approach, there will now be (depending whether or not the system Python is used) both:
SAGE_LOCAL/lib/python3.7/site-packages
SAGE_LOCAL/lib/sage/venv/sage/lib/python3.7/site-packages
If SAGE_LOCAL
itself (if it exists) were a virtualenv already that would simplify matters greatly.
comment:34 in reply to: ↑ 33 ; follow-up: ↓ 36 Changed 3 years ago by
Replying to embray:
Or, at the very least, what I've suggested in the past was to make
$SAGE_LOCAL
itself a virtualenv, as virtualenvs already have a sort-of root filesystem, including among other thingslib/pythonX.Y/site-packages
.
No, the present ticket explicitly avoids that.
The reason is that venv
has no documented mechanism for adding non-Python stuff to it, in particular to its activation scripts.
comment:35 in reply to: ↑ 33 Changed 3 years ago by
Replying to embray:
With this approach, there will now be (depending whether or not the system Python is used) both:
SAGE_LOCAL/lib/python3.7/site-packages
SAGE_LOCAL/lib/sage/venv/sage/lib/python3.7/site-packages
That's correct. In a fresh install, the first one will be empty.
In an install upgraded from earlier versions, there will be packages in both. When there's an update, the old version if removed from the first one and the new version is added to the second one.
comment:36 in reply to: ↑ 34 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
Or, at the very least, what I've suggested in the past was to make
$SAGE_LOCAL
itself a virtualenv, as virtualenvs already have a sort-of root filesystem, including among other thingslib/pythonX.Y/site-packages
.No, the present ticket explicitly avoids that. The reason is that
venv
has no documented mechanism for adding non-Python stuff to it, in particular to its activation scripts.
There's no reason you can't--I've often install C libraries into a virtualenv by setting the path to it in ./configure --prefix
.
comment:37 follow-up: ↓ 39 Changed 3 years ago by
I really hate to be a pain in the ass because I know you put a lot of work into this, but I really don't get it... It really seems complicated and messy to both activate the Sage environment, and then within that activate a virtualenv. I don't think that's what I ever had in mind when I suggested using a virtualenv for Sage (or if I did, I take it back...)
I'd still like to know what problem this is solving. You say that it's required for #27824, but why? What specific problems related to #27824 is it solving? Because most likely there's another way.
comment:38 follow-up: ↓ 40 Changed 3 years ago by
comment:39 in reply to: ↑ 37 Changed 3 years ago by
Replying to embray:
I don't think that's what I ever had in mind when I suggested using a virtualenv for Sage
Don't worry. I'm prepared to take full credit for this specific version of the idea, if necessary.
comment:40 in reply to: ↑ 38 ; follow-up: ↓ 41 Changed 3 years ago by
Replying to dimpase:
Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.
How? Why? There's really nothing that magical about virtualenvs. It's just an alternate install prefix plus (as of Python 3 and the venv module) some small support bits in the interpreter to report a different sys.prefix
, etc.
comment:41 in reply to: ↑ 40 ; follow-up: ↓ 42 Changed 3 years ago by
Replying to embray:
Replying to dimpase:
Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.
How? Why? There's really nothing that magical about virtualenvs. It's just an alternate install prefix plus (as of Python 3 and the venv module) some small support bits in the interpreter to report a different
sys.prefix
, etc.
Part of the magic is to use the standard instead of project-specific tricks.
comment:42 in reply to: ↑ 41 Changed 3 years ago by
Replying to mkoeppe:
Replying to embray:
Replying to dimpase:
Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.
How? Why? There's really nothing that magical about virtualenvs. It's just an alternate install prefix plus (as of Python 3 and the venv module) some small support bits in the interpreter to report a different
sys.prefix
, etc.Part of the magic is to use the standard instead of project-specific tricks.
What "tricks" are you avoiding?
comment:43 Changed 3 years ago by
- Commit changed from 27d945a8d41354234c123f02e2e1d41d287a7541 to f2af47873732f2142fc4b6b3bf6b4aed94a2503a
Branch pushed to git repo; I updated commit sha1. New commits:
08ec551 | build/make/deps: Make python3_venv a .PHONY target
|
b2ab6f3 | src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
|
7a43544 | src/sage/env.py, envconfig.py.in: Add MAXIMA; src/sage/interfaces/maxima.py: Use it
|
f2af478 | Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv
|
comment:44 in reply to: ↑ 21 Changed 3 years ago by
comment:45 Changed 3 years ago by
comment:46 Changed 3 years ago by
- Commit changed from f2af47873732f2142fc4b6b3bf6b4aed94a2503a to 199a72ed50ae52a1024872e1c417fb3919a11eb8
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
6b28ac1 | In a python3 build, install all Python packages into a venv
|
981f65e | sage-pip-install: Use PYTHON=sage-python23
|
a6854bd | _get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
|
11ada01 | build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg
|
3bd10e4 | Add cryptominisat patch for virtual envs
|
da13a9e | moved patch to the right place
|
c9d067a | src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
|
199a72e | src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL
|
comment:47 Changed 3 years ago by
comment:48 Changed 3 years ago by
- Status changed from needs_review to needs_work
- Work issues set to Redo on top of current #29022; #29038 instead of #29022
comment:49 Changed 3 years ago by
- Dependencies changed from #29002, #29022, #29025 to #29002, #29022, #29025, #29038
comment:50 Changed 3 years ago by
I would be interested in your comments on #29032. By making $SAGE_LOCAL
itself into a virtualenv--which it already essentially is in all other respects--no other weird hacks are needed.
Other than some things like the OSX fixes I don't see what this solves that my branch doesn't.
comment:51 Changed 2 years ago by
- Commit changed from 199a72ed50ae52a1024872e1c417fb3919a11eb8 to 208c7e68c24c1fe896a5e6573c6aae5399056431
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
980f444 | In a python3 build, install all Python packages into a venv
|
90bf090 | sage-pip-install: Use PYTHON=sage-python23
|
9edf068 | _get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
|
ead2d0f | Add cryptominisat patch for virtual envs
|
56e3daa | src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
|
208c7e6 | src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL
|
comment:52 Changed 2 years ago by
- Dependencies #29002, #29022, #29025, #29038 deleted
- Work issues Redo on top of current #29022; #29038 instead of #29022 deleted
comment:53 Changed 2 years ago by
- Dependencies set to #29362
comment:54 Changed 2 years ago by
- Description modified (diff)
- Milestone changed from sage-9.1 to sage-wishlist
- Summary changed from In a python3 build, install all Python packages into a venv to In a python3 build, install all Python packages into a venv different from SAGE_LOCAL
comment:55 Changed 2 years ago by
- Cc jhpalmieri added
- Dependencies #29362 deleted
- Description modified (diff)
- Milestone changed from sage-wishlist to sage-9.3
- Summary changed from In a python3 build, install all Python packages into a venv different from SAGE_LOCAL to Support installation of Python packages into separate venvs depending on the python version
comment:56 Changed 2 years ago by
- Description modified (diff)
comment:57 Changed 2 years ago by
- Description modified (diff)
comment:58 Changed 2 years ago by
- Description modified (diff)
comment:59 Changed 2 years ago by
- Dependencies set to #22731
comment:60 Changed 2 years ago by
- Description modified (diff)
comment:61 Changed 23 months ago by
- Description modified (diff)
comment:62 Changed 23 months ago by
- Dependencies changed from #22731 to #22731, #30719
- Description modified (diff)
comment:63 Changed 23 months ago by
- Dependencies changed from #22731, #30719 to #22731, #30719, #30657
comment:64 Changed 23 months ago by
- Commit changed from 208c7e68c24c1fe896a5e6573c6aae5399056431 to 1ead553c9407060c3459a92138b8448b5aa2fd3b
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
11ce883 | Add more build/pkgs/*/install-requires.txt
|
85b2dcf | for a in *; do if [ -d $a -a ! -r $a/requirements.txt -a ! -r $a/install-requires.txt ]; then if grep -q -i sdh_pip $a/spkg-install.in >&/dev/null ; then ( if [ -r $a/package-version.txt ]; then echo "$a >=$(sed s/[.]p[0-9]$// $a/package-version.txt)"; else echo "$a"; fi) > $a/install-requires.txt && git add $a/install-requires.txt; fi; fi; done
|
f9ac70e | Add install-requires.txt for setuptools, pip
|
0635f61 | Merge branch 't/30719/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages' into t/29013/separate_venvs
|
082d3c0 | build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV
|
bd0fd2f | Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30657/fix_up__sage__p_
|
2e20f23 | build/bin/sage-spkg: Move SAGE_SPKG_{INST,SCRIPTS,WHEELS} setting here from build/bin/install, make them relative to SAGE_INST_LOCAL
|
be2417c | Merge tag '9.2.beta14' into t/30657/fix_up__sage__p_
|
8ee7ee2 | build/make/install: Set SAGE_SPKG_WHEELS also here so that script packages still work
|
1ead553 | Merge branch 't/30657/fix_up__sage__p_' into t/29013/separate_venvs
|
comment:66 Changed 23 months ago by
- Dependencies changed from #22731, #30719, #30657 to #22731, #30719, #30657, #30622
comment:67 Changed 23 months ago by
- Commit changed from 1ead553c9407060c3459a92138b8448b5aa2fd3b to 6297dd72e65961bf68c16eb17d6a3787c24f07df
Branch pushed to git repo; I updated commit sha1. New commits:
be554b0 | build/bin/{sage-spkg,sage-dist-helpers}: Use SAGE_INST_LOCAL
|
8ca2d23 | Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix
|
4fbf79f | build/bin/sage-spkg: Move SAGE_SPKG_{INST,SCRIPTS,WHEELS} setting here from build/bin/install, make them relative to SAGE_INST_LOCAL
|
8fc7f9c | Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix
|
0f1b7aa | build/make/install: Do not set SAGE_SPKG_INST here; set it in build/make/Makefile.in
|
1baf8ea | WIP
|
a713c39 | Merge branch 't/30657/fix_up__sage__p_' into t/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix
|
6297dd7 | Merge branch 't/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix' into t/29013/separate_venvs
|
comment:68 Changed 23 months ago by
- Dependencies changed from #22731, #30719, #30657, #30622 to #30719, #30657, #30622 | #22731, #30724
When sage-env-config
is sourced, it overrides PYTHON_FOR_VENV
.
This will be addressed in #30724.
comment:69 Changed 23 months ago by
- Commit changed from 6297dd72e65961bf68c16eb17d6a3787c24f07df to 5673cd008d7e09f3f2b2208adc998b06c767ead4
Branch pushed to git repo; I updated commit sha1. New commits:
91b4c0f | build/make/Makefile.in: Set PYTHON_FOR_VENV here, do not rely on sage-env to set it
|
1955fad | src/bin/sage-env: Set MACOSX_DEPLOYMENT_TARGET based on config variable SAGE_MACOSX_DEPLOYMENT_TARGET
|
4bfa0ea | src/bin/sage-env-config.in: Do not set PYTHON_FOR_VENV
|
5673cd0 | Merge branch 't/30724/src_bin_sage_env__src_bin_sage_env_config_in__remove_python_for_venv' into t/29013/separate_venvs
|
comment:70 Changed 23 months ago by
- Dependencies changed from #30719, #30657, #30622 | #22731, #30724 to #30719, #30657, #30622, #30724 | #22731
comment:71 Changed 23 months ago by
- Commit changed from 5673cd008d7e09f3f2b2208adc998b06c767ead4 to cc71adae894d12cd902b5cecbd98118d70d283e5
Branch pushed to git repo; I updated commit sha1. New commits:
cc71ada | WIP
|
comment:72 Changed 23 months ago by
- Dependencies changed from #30719, #30657, #30622, #30724 | #22731 to #30719, #30657, #30622, #30724 | #22731, #30731
comment:73 Changed 23 months ago by
- Commit changed from cc71adae894d12cd902b5cecbd98118d70d283e5 to 9b93425dad001fe468e0f1551a9b55c0f8289d62
comment:74 Changed 21 months ago by
- Dependencies changed from #30719, #30657, #30622, #30724 | #22731, #30731 to #30719, #30657, #30622, #30724 | #22731, #30731, #30896
- Work issues set to refactor through #30896
comment:75 Changed 21 months ago by
- Description modified (diff)
comment:76 Changed 21 months ago by
- Description modified (diff)
comment:77 Changed 21 months ago by
- Keywords sd111 added
Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111
comment:78 Changed 20 months ago by
- Description modified (diff)
comment:79 Changed 18 months ago by
- Dependencies #30719, #30657, #30622, #30724 | #22731, #30731, #30896 deleted
- Work issues changed from refactor through #30896 to refactor through #31362, #30896
comment:80 Changed 18 months ago by
- Milestone changed from sage-9.3 to sage-9.4
comment:81 Changed 18 months ago by
- Dependencies set to #31362
- Work issues changed from refactor through #31362, #30896 to refactor through #31362
comment:82 Changed 18 months ago by
- Commit changed from 9b93425dad001fe468e0f1551a9b55c0f8289d62 to 2f6bb57173ba5b067334be57a83089f2e3ef6951
Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:
ea182d7 | Copy changes from build/pkgs/sagelib/src to src
|
a1a10b9 | src/VERSION.txt: New
|
5697335 | src/setup.cfg: Add license_file=LICENSE.txt
|
deb9eb3 | Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
|
7ad4c0e | Merge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
|
27b589b | Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
|
8796008 | Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
|
d6831b1 | build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
|
61f6ba6 | Makefile: Add top-level targets build-local, build-venv
|
2f6bb57 | build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV
|
comment:83 Changed 18 months ago by
- Commit changed from 2f6bb57173ba5b067334be57a83089f2e3ef6951 to 3c731b8014a1ab6e323a87c73e6da1da160fd4a7
Branch pushed to git repo; I updated commit sha1. New commits:
3c731b8 | build/make/Makefile.in: Rework using tree_... variables
|
comment:84 Changed 18 months ago by
- Commit changed from 3c731b8014a1ab6e323a87c73e6da1da160fd4a7 to d04b08b61ba7e86e75da625a30717b9809606cb7
Branch pushed to git repo; I updated commit sha1. New commits:
d04b08b | build/make/Makefile.in: Define SAGE_VENV earlier
|
comment:85 Changed 18 months ago by
- Work issues changed from refactor through #31362 to AC_ARG_WITH
comment:86 Changed 18 months ago by
- Description modified (diff)
comment:87 Changed 18 months ago by
- Commit changed from d04b08b61ba7e86e75da625a30717b9809606cb7 to 306ddefd5806daa5ce8a720a1480c28cea76c5f6
Branch pushed to git repo; I updated commit sha1. New commits:
306ddef | WIP configure --with-sage-venv
|
comment:88 Changed 17 months ago by
- Commit changed from 306ddefd5806daa5ce8a720a1480c28cea76c5f6 to 8cc0b3ca43d4c9f852d86079fa6edb1d6aa4cf14
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
bf6e5f8 | build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV
|
832f96a | build/make/Makefile.in: Rework using tree_... variables
|
3436ad6 | build/make/Makefile.in: Define SAGE_VENV earlier
|
8cc0b3c | WIP configure --with-sage-venv
|
comment:89 Changed 17 months ago by
- Dependencies #31362 deleted
comment:90 Changed 17 months ago by
- Commit changed from 8cc0b3ca43d4c9f852d86079fa6edb1d6aa4cf14 to 2202f842b7f7bf0a9f83d9dcdb07f57c2002dfd2
comment:91 Changed 17 months ago by
- Status changed from needs_work to needs_review
- Work issues AC_ARG_WITH deleted
comment:92 Changed 17 months ago by
- Commit changed from 2202f842b7f7bf0a9f83d9dcdb07f57c2002dfd2 to 92f1a1bba94a5dd6a4c9458438c39b781b101ead
comment:93 Changed 17 months ago by
- Description modified (diff)
comment:94 Changed 17 months ago by
- Summary changed from Support installation of Python packages into separate venvs depending on the python version to Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL
comment:95 Changed 17 months ago by
- Commit changed from 92f1a1bba94a5dd6a4c9458438c39b781b101ead to 9dcded5e3188eec1c4f0d1a81016783f506708fd
comment:96 Changed 17 months ago by
With ./configure --with-python=/usr/bin/python3 --with-sage-venv="`pwd`/local/var/lib/sage/venv/python3.8"
, I'm getting some doctest failures:
sage -t --long --warn-long 101.2 --random-seed=0 src/sage/misc/package.py # 5 doctests failed
(because it doesn't find some Python packages). Also
sage -t --long --warn-long 101.2 --random-seed=0 src/sage_setup/find.py # 2 doctests failed sage -t --long --warn-long 101.2 --random-seed=0 src/sage_setup/clean.py # 1 doctest failed
Details:
File "src/sage_setup/find.py", line 365, in sage_setup.find.installed_files_by_module Failed example: f1 Expected: 'sage/structure/__init__.py' Got: '/Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.8/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc' ********************************************************************** File "src/sage_setup/find.py", line 367, in sage_setup.find.installed_files_by_module Failed example: f2 Expected: 'sage/structure/....pyc' Got: 'sage/structure/__init__.py' ********************************************************************** File "src/sage_setup/clean.py", line 96, in sage_setup.clean._find_stale_files Failed example: for f in stale_iter: if f.endswith(skip_extensions): continue if '/ext_data/' in f: continue print('Found stale file: ' + f) Expected nothing Got: Found stale file: /Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.8/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc ... [followed by many many lines of "stale" files]
comment:97 Changed 17 months ago by
- Status changed from needs_review to needs_work
The ones in find
and clean
are I think #31314.
For the failure in package
I know what to do
comment:98 Changed 17 months ago by
- Commit changed from 9dcded5e3188eec1c4f0d1a81016783f506708fd to 6e0c19ec5bf263ea75dffa0d5d7332daf37e6d6e
Branch pushed to git repo; I updated commit sha1. New commits:
6e0c19e | sage.misc.package.installed_packages: After SAGE_SPKG_INST, also go through SAGE_VENV_SPKG_INST
|
comment:99 Changed 17 months ago by
- Status changed from needs_work to needs_review
comment:100 Changed 17 months ago by
- Status changed from needs_review to needs_work
The sage_setup doctest failures have vanished, but I am seeing different problems with misc/package.py
:
********************************************************************** File "src/sage/misc/package.py", line 554, in sage.misc.package.package_manifest Failed example: sagetex_manifest = package_manifest('sagetex') # optional - build Exception raised: Traceback (most recent call last): File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute exec(compiled, globs) File "<doctest sage.misc.package.package_manifest[1]>", line 1, in <module> sagetex_manifest = package_manifest('sagetex') # optional - build File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/misc/package.py", line 570, in package_manifest with open(stamp_file) as f: FileNotFoundError: [Errno 2] No such file or directory: '/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/installed/sagetex-3.5' ********************************************************************** File "src/sage/misc/package.py", line 555, in sage.misc.package.package_manifest Failed example: sagetex_manifest['package_name'] == 'sagetex' # optional - build Exception raised: Traceback (most recent call last): File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute exec(compiled, globs) File "<doctest sage.misc.package.package_manifest[2]>", line 1, in <module> sagetex_manifest['package_name'] == 'sagetex' # optional - build NameError: name 'sagetex_manifest' is not defined ********************************************************************** File "src/sage/misc/package.py", line 557, in sage.misc.package.package_manifest Failed example: 'files' in sagetex_manifest # optional - build Exception raised: Traceback (most recent call last): File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute exec(compiled, globs) File "<doctest sage.misc.package.package_manifest[3]>", line 1, in <module> 'files' in sagetex_manifest # optional - build NameError: name 'sagetex_manifest' is not defined **********************************************************************
I guess the issue is that manifests are now written to two different places, depending on whether it's a Python package or not.
comment:101 Changed 17 months ago by
Right, I forgot about this function. It will need a slightly generalized interface
comment:102 Changed 17 months ago by
- Commit changed from 6e0c19ec5bf263ea75dffa0d5d7332daf37e6d6e to d68e86144b4a37600654899d2ab655ae55fb4378
comment:103 Changed 17 months ago by
- Status changed from needs_work to needs_review
comment:104 follow-up: ↓ 105 Changed 17 months ago by
That fixes the package.py
doctest for me, thank you. I'm not sure I fully understand how venvs are supposed to work. If I do ./configure --with-sage-venv=... && make
and then separately do ./configure && make
, I should get two builds (as far as Python packages are concerned), which I could conceivably switch between, right? Now if I do ./configure && make pysingular
, then I can do import PySingular
, but I switch to the other venv, I won't be able to. However, 'pysingular' in installed_packages()
returns True
regardless of whether I'm using the venv, since it searches both installation directories.
(a) Am I understanding the situation correctly? and
(b) is it worth trying to fix?
comment:105 in reply to: ↑ 104 Changed 17 months ago by
Replying to jhpalmieri:
I'm not sure I fully understand how venvs are supposed to work. If I do
./configure --with-sage-venv=... && make
and then separately do./configure && make
, I should get two builds (as far as Python packages are concerned), which I could conceivably switch between, right?
Yes, that's right. Separate for Python packages, sharing the non-Python packages, which are only in SAGE_LOCAL.
Now if I do
./configure && make pysingular
, then I can doimport PySingular
, but I switch to the other venv, I won't be able to. However,'pysingular' in installed_packages()
returnsTrue
regardless of whether I'm using the venv, since it searches both installation directories.
OK, thanks for catching this. Yes, the sage.misc.package
code does not handle this situation correctly. I would simply suggest to not mix builds that use with --with-sage-venv=...
with builds that don't. You can switch between several venvs in this way.
comment:106 Changed 17 months ago by
- Reviewers set to John Palmieri
- Status changed from needs_review to positive_review
Great, let's merge it, then.
comment:107 Changed 17 months ago by
Thank you!
comment:108 Changed 15 months ago by
- Branch changed from public/29013-use-py3-venv to d68e86144b4a37600654899d2ab655ae55fb4378
- Resolution set to fixed
- Status changed from positive_review to closed
A simple change. Compiles from a fresh checkout and runs without problems on macOS.
New commits:
In a python3 build, install all Python packages into a venv
sage-pip-install: Use PYTHON=sage-python23