Add configure option --with-sage-venv=SAGE_VENV to create venv there instead of in SAGE_LOCAL

[mkoeppe]

#27824 made $SAGE_LOCAL a venv if a suitable system python3 is found.

In this ticket, we add support for installing builds of Sage with several python versions, sharing the non-Python packages in SAGE_LOCAL:

• keeping install records for Python packages (recognized by having install-requires.txt or requirements.txt) in $SAGE_VENV/var/lib/sage/{installed,scripts,wheels}, separate from those of non-Python packages ($SAGE_LOCAL/var/lib/sage/{installed,scripts})
• likewise for $SAGE_LOCAL/var/tmp/sage/build
• here $SAGE_VENV defaults to $SAGE_LOCAL, but can be overridden to an arbitrary directory that will be used as the wheel-building venv, for example $SAGE_LOCAL/var/lib/sage/venv/$PYTHON_TAG.

Configuration:

• This is activated by running, for example,

./configure --with-python=/usr/bin/python3.8 --with-sage-venv="$SAGE_LOCAL/var/lib/sage/venv/python3.8".

As of this ticket, we are able to establish the venv in an arbitrary configured directory instead of SAGE_LOCAL. This is already useful for #31396 to create a wheel-building venv that will not be packaged as part of SAGE_LOCAL.

In follow-up tickets, we gain the full functionality that facilitates testing with several Python versions without having to rebuild the Sage distribution. This depends on:

• #30534 Repackage pynac as a pip-installable package

Follow-ups:

• We also want to support ./configure --with-sage-venv=no which would suppress making any venv (#30896)

• Support the tricky case: When system python3 is not in use and SAGE_VENV != SAGE_LOCAL, then both a real $SAGE_LOCAL/bin/python3 needs to be built and a venv in SAGE_VENV created... or should python3 be installed in $SAGE_VENV?)

• Optional: A file build/pkgs/SPKG/trees could override the install tree determination - this would allow us to for example install a package such as jupyter_core both into something like SAGE_NOTEBOOK_VENV and SAGE_VENV...

[mkoeppe]

A simple change. Compiles from a fresh checkout and runs without problems on macOS.

---

New commits:

​1363425	In a python3 build, install all Python packages into a venv
​a394268	sage-pip-install: Use PYTHON=sage-python23

[mkoeppe]

In the branch #27824, the venv is created by slightly more complicated code in build/make/Makefile. This complexity is not necessary for the present ticket.

[dimpase]

Does this ticket make sense separately, or only as part of #27824 ?

[mkoeppe]

#27824 is the use case that I have in mind. This ticket is a less intrusive change and should go through the betas first.

[jhpalmieri]

I tried building with this branch on OS X, and I got doctest failures:

sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/matrix/matrix2.pyx  # 2 doctests failed
sage -t --long src/sage/finance/time_series.pyx  # 5 doctests failed
sage -t --long src/sage/misc/trace.py  # 4 doctests failed
sage -t --long src/sage/env.py  # 1 doctest failed
sage -t --long src/sage/repl/configuration.py  # 1 doctest failed
sage -t --long src/sage/schemes/affine/affine_homset.py  # 1 doctest failed

For example

File "src/sage/plot/plot.py", line 521, in sage.plot.plot
Failed example:
    os.system("sage -c \"if 'matplotlib' in sys.modules: sys.exit(1)\"") # long time
Expected:
    0
Got:
    Traceback (most recent call last):
      File "/Users/jpalmier/Desktop/Sage/sage_builds/TESTING/sage-9.1.beta0/local/bin/sage-eval", line 4, in <module>
        from sage.all import *
    ModuleNotFoundError: No module named 'sage'
    256

and

**********************************************************************
File "src/sage/matrix/matrix2.pyx", line 1127, in sage.matrix.matrix2.Matrix.pseudoinverse
Failed example:
    M.pseudoinverse()  # tol 1e-15
Expected:
    [0.0620518477661335 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107557 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Got:
    [0.0620518477661334 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107556 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Tolerance exceeded in 2 of 9:
    0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15
    0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15
**********************************************************************
File "src/sage/matrix/matrix2.pyx", line 1131, in sage.matrix.matrix2.Matrix.pseudoinverse
Failed example:
    M.pseudoinverse(algorithm="numpy")  # tol 1e-15
Expected:
    [0.0620518477661335 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107557 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Got:
    [0.0620518477661334 0.0206839492553778 0.0124103695532267]
    [ 0.124103695532267 0.0413678985107556 0.0248207391064534]
    [ 0.186155543298400 0.0620518477661335 0.0372311086596801]
Tolerance exceeded in 2 of 9:
    0.0620518477661335 vs 0.0620518477661334, tolerance 2e-15 > 1e-15
    0.0413678985107557 vs 0.0413678985107556, tolerance 3e-15 > 1e-15
**********************************************************************
1 item had failures:
   2 of  25 in sage.matrix.matrix2.Matrix.pseudoinverse
    [2472 tests, 2 failures, 9.06 s]

[mkoeppe]

Thanks for testing!

[mkoeppe]

Replying to jhpalmieri:

sage -t --long src/sage/matrix/matrix2.pyx # 2 doctests failed

Not sure what's causing the numerical failures.

sage -t --long src/sage/env.py # 1 doctest failed

For this one, I have created #29022.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d4fbe40

_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bba08ec	Create module src/sage/env_config.py from src/sage/env_config.py.in, defining SAGE_LOCAL
​d1779cc	Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv

[mkoeppe]

Replying to mkoeppe:

sage -t --long src/sage/env.py # 1 doctest failed

For this one, I have created #29022.

Fixed.

[mkoeppe]

The numerical problems seems to happen because numpy does not find BLAS.

[mkoeppe]

Actually, it does not find openblas or atlas, but it does find the "Accelerate framework":

accelerate_info:
  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​85147f9	build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg
​93bae6f	Merge branch 't/29025/numpy__site_cfg_needs_a__default__section' into t/29013/public/29013-use-py3-venv

[mkoeppe]

Merged #29025, fixes the numerical doctests.

[dimpase]

I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)

sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py
**********************************************************************
File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat
Failed example:
    solver = CryptoMiniSat()                                  # optional - cryptominisat
Exception raised:
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__
        from pycryptosat import Solver
    ModuleNotFoundError: No module named 'pycryptosat'

I wonder whether it uses a convoluted way to create python extensions, which got broken.

[dimpase]

oops, comment:18 belongs to #27824

[embray]

Minor bikeshed, but why the long path "lib/sage/venv/sage"? What about something like just "lib/sage-X.Y"; analogously to "lib/python-X.Y"? We've never had a "lib/sage" before in the first place, so I don't see any immediate need to make the path so much more complicated.

[mkoeppe]

Replying to embray:

Minor bikeshed, but why the long path "lib/sage/venv/sage"? What about something like just "lib/sage-X.Y"; analogously to "lib/python-X.Y"? We've never had a "lib/sage" before in the first place, so I don't see any immediate need to make the path so much more complicated.

1) "venv" has to be in the path to indicate that it's a venv in the technical sense and not a random place with invitation to install stuff in it.

2) Definitely no versioning.

3) I suppose I could drop the final "sage", making it lib/sage/venv

[mkoeppe]

Replying to dimpase:

I see a problem with cryptominisat spkg on this branch (which otherwise builds and passes most tests (not related to this packages) on Gentoo)

sage -t --warn-long 88.1 src/sage/sat/solvers/cryptominisat.py
**********************************************************************
File "src/sage/sat/solvers/cryptominisat.py", line 49, in sage.sat.solvers.cryptominisat.CryptoMiniSat
Failed example:
    solver = CryptoMiniSat()                                  # optional - cryptominisat
Exception raised:
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/sage/venv/sage/lib/python3.7/site-packages/sage/sat/solvers/cryptominisat.py", line 69, in __init__
        from pycryptosat import Solver
    ModuleNotFoundError: No module named 'pycryptosat'

I wonder whether it uses a convoluted way to create python extensions, which got broken.

Upstream merged pull request ​https://github.com/msoos/cryptominisat/pull/551 to take care of a virtualenv installation.

[mkoeppe]

I haven't tested whether this patch helps for our venv. The patch is not in the current release 5.6.8 (which we have).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2ed5d07

Add cryptominisat patch for virtual envs

[mkoeppe]

Try with this patch please. I can't compile cryptominisat because of an unrelated reason.

[dimpase]

patch landed into a wrong place, it should be one level down, in patches/

testing now, after correction of this.

[mkoeppe]

Sorry, thanks.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​27d945a

moved patch to the right place

[dimpase]

the patch worked, it's all good with tests now.

[mkoeppe]

Thank you!

[embray]

I might have missed some discussion elsewhere, but I still don't see the advantage of this extra complication over having the sage environment just extend sys.path where necessary. I say this as a big fan of virtualenvs in general, and having suggested doing something like this in the past. But the more I think about it the less I see any advantage in this case over having an additional sys.path entry.

[mkoeppe]

Replying to embray:

I say this as a big fan of virtualenvs in general, and having suggested doing something like this in the past.

Yes, of course this ticket was inspired by various insights and discussions shared by you and others on trac tickets.

But the more I think about it the less I see any advantage in this case over having an additional sys.path entry.

Previously, with our python2 build, there was no clear way to do it. But now with python 3 there is a standard and clean tool: that's venv. Installing into it is supported by standard tools.

[embray]

Or, at the very least, what I've suggested in the past was to make $SAGE_LOCAL itself a virtualenv, as virtualenvs already have a sort-of root filesystem, including among other things lib/pythonX.Y/site-packages.

With this approach, there will now be (depending whether or not the system Python is used) both:

• SAGE_LOCAL/lib/python3.7/site-packages
• SAGE_LOCAL/lib/sage/venv/sage/lib/python3.7/site-packages

If SAGE_LOCAL itself (if it exists) were a virtualenv already that would simplify matters greatly.

[mkoeppe]

Replying to embray:

Or, at the very least, what I've suggested in the past was to make $SAGE_LOCAL itself a virtualenv, as virtualenvs already have a sort-of root filesystem, including among other things lib/pythonX.Y/site-packages.

No, the present ticket explicitly avoids that. The reason is that venv has no documented mechanism for adding non-Python stuff to it, in particular to its activation scripts.

[mkoeppe]

Replying to embray:

With this approach, there will now be (depending whether or not the system Python is used) both:

• SAGE_LOCAL/lib/python3.7/site-packages
• SAGE_LOCAL/lib/sage/venv/sage/lib/python3.7/site-packages

That's correct. In a fresh install, the first one will be empty.

In an install upgraded from earlier versions, there will be packages in both. When there's an update, the old version if removed from the first one and the new version is added to the second one.

[embray]

Replying to mkoeppe:

Replying to embray:

Or, at the very least, what I've suggested in the past was to make $SAGE_LOCAL itself a virtualenv, as virtualenvs already have a sort-of root filesystem, including among other things lib/pythonX.Y/site-packages.

No, the present ticket explicitly avoids that. The reason is that venv has no documented mechanism for adding non-Python stuff to it, in particular to its activation scripts.

There's no reason you can't--I've often install C libraries into a virtualenv by setting the path to it in ./configure --prefix.

[embray]

I really hate to be a pain in the ass because I know you put a lot of work into this, but I really don't get it... It really seems complicated and messy to both activate the Sage environment, and then within that activate a virtualenv. I don't think that's what I ever had in mind when I suggested using a virtualenv for Sage (or if I did, I take it back...)

I'd still like to know what problem this is solving. You say that it's required for #27824, but why? What specific problems related to #27824 is it solving? Because most likely there's another way.

[dimpase]

Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.

[mkoeppe]

Replying to embray:

I don't think that's what I ever had in mind when I suggested using a virtualenv for Sage

Don't worry. I'm prepared to take full credit for this specific version of the idea, if necessary.

[embray]

Replying to dimpase:

Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.

How? Why? There's really nothing that magical about virtualenvs. It's just an alternate install prefix plus (as of Python 3 and the venv module) some small support bits in the interpreter to report a different sys.prefix, etc.

[mkoeppe]

Replying to embray:

Replying to dimpase:

Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.

How? Why? There's really nothing that magical about virtualenvs. It's just an alternate install prefix plus (as of Python 3 and the venv module) some small support bits in the interpreter to report a different sys.prefix, etc.

Part of the magic is to use the standard instead of project-specific tricks.

[embray]

Replying to mkoeppe:

Replying to embray:

Replying to dimpase:

Well, the separation of #29013 and #27824 is artificial, but otherwise, they do the job. venv removes an extra complication of sagelib being very far from pip-installable.

How? Why? There's really nothing that magical about virtualenvs. It's just an alternate install prefix plus (as of Python 3 and the venv module) some small support bits in the interpreter to report a different sys.prefix, etc.

Part of the magic is to use the standard instead of project-specific tricks.

What "tricks" are you avoiding?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​08ec551	build/make/deps: Make python3_venv a .PHONY target
​b2ab6f3	src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
​7a43544	src/sage/env.py, envconfig.py.in: Add MAXIMA; src/sage/interfaces/maxima.py: Use it
​f2af478	Merge branch 't/29022/create_module_src_sage_env_config_py_from_src_sage_env_config_py_in__defining_variables_for_use_in_sage_env' into t/29013/public/29013-use-py3-venv

[mkoeppe]

Replying to mkoeppe:

Replying to embray:

Minor bikeshed, but why the long path "lib/sage/venv/sage"? ....

3) I suppose I could drop the final "sage", making it lib/sage/venv

Done.

[mkoeppe]

I've also merged in a few changes from #29032 (Erik's version of python spkg-configure / venv), and the current #29022.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6b28ac1	In a python3 build, install all Python packages into a venv
​981f65e	sage-pip-install: Use PYTHON=sage-python23
​a6854bd	_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
​11ada01	build/pkgs/numpy/lapack_conf.py: Add a [DEFAULT] section to site.cfg
​3bd10e4	Add cryptominisat patch for virtual envs
​da13a9e	moved patch to the right place
​c9d067a	src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
​199a72e	src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL

[embray]

I would be interested in your comments on #29032. By making $SAGE_LOCAL itself into a virtualenv--which it already essentially is in all other respects--no other weird hacks are needed.

Other than some things like the OSX fixes I don't see what this solves that my branch doesn't.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​980f444	In a python3 build, install all Python packages into a venv
​90bf090	sage-pip-install: Use PYTHON=sage-python23
​9edf068	_get_shared_lib_filename: Do not assume Python sysconfig paths are in SAGE_LOCAL
​ead2d0f	Add cryptominisat patch for virtual envs
​56e3daa	src/bin/sage-env: Simplify the venv path to $SAGE_LOCAL/lib/sage/venv
​208c7e6	src/sage/env.py: Rework _get_shared_lib_filename, fix for Cygwin: it would privilege system DLLs over DLLs in SAGE_LOCAL

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​11ce883	Add more build/pkgs/*/install-requires.txt
​85b2dcf	for a in *; do if [ -d $a -a ! -r $a/requirements.txt -a ! -r $a/install-requires.txt ]; then if grep -q -i sdh_pip $a/spkg-install.in >&/dev/null ; then ( if [ -r $a/package-version.txt ]; then echo "$a >=$(sed s/[.]p[0-9]$// $a/package-version.txt)"; else echo "$a"; fi) > $a/install-requires.txt && git add $a/install-requires.txt; fi; fi; done
​f9ac70e	Add install-requires.txt for setuptools, pip
​0635f61	Merge branch 't/30719/add_build_pkgs_spkg_install_requires_txt_for_all_python_packages' into t/29013/separate_venvs
​082d3c0	build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV
​bd0fd2f	Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30657/fix_up__sage__p_
​2e20f23	build/bin/sage-spkg: Move SAGE_SPKG_{INST,SCRIPTS,WHEELS} setting here from build/bin/install, make them relative to SAGE_INST_LOCAL
​be2417c	Merge tag '9.2.beta14' into t/30657/fix_up__sage__p_
​8ee7ee2	build/make/install: Set SAGE_SPKG_WHEELS also here so that script packages still work
​1ead553	Merge branch 't/30657/fix_up__sage__p_' into t/29013/separate_venvs

[mkoeppe]

Replaced the branch

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​be554b0	build/bin/{sage-spkg,sage-dist-helpers}: Use SAGE_INST_LOCAL
​8ca2d23	Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix
​4fbf79f	build/bin/sage-spkg: Move SAGE_SPKG_{INST,SCRIPTS,WHEELS} setting here from build/bin/install, make them relative to SAGE_INST_LOCAL
​8fc7f9c	Merge branch 't/29500/install_all_python_packages_via_pip_wheel__create_pep_503_simple_repository_for_wheels' into t/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix
​0f1b7aa	build/make/install: Do not set SAGE_SPKG_INST here; set it in build/make/Makefile.in
​1baf8ea	WIP
​a713c39	Merge branch 't/30657/fix_up__sage__p_' into t/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix
​6297dd7	Merge branch 't/30622/build_bin_sage_spkg__allow_overriding_sage_local_by_sage_spkg_prefix' into t/29013/separate_venvs

[mkoeppe]

When sage-env-config is sourced, it overrides PYTHON_FOR_VENV. This will be addressed in #30724.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​91b4c0f	build/make/Makefile.in: Set PYTHON_FOR_VENV here, do not rely on sage-env to set it
​1955fad	src/bin/sage-env: Set MACOSX_DEPLOYMENT_TARGET based on config variable SAGE_MACOSX_DEPLOYMENT_TARGET
​4bfa0ea	src/bin/sage-env-config.in: Do not set PYTHON_FOR_VENV
​5673cd0	Merge branch 't/30724/src_bin_sage_env__src_bin_sage_env_config_in__remove_python_for_venv' into t/29013/separate_venvs

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cc71ada

WIP

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​332c723	build/bin/sage-pip-install, build/bin/sage-python23: Just call python3
​9b93425	Working version for normal packages

[mkoeppe]

Hoping we can make progress on this ticket this week - ​https://wiki.sagemath.org/days111

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​ea182d7	Copy changes from build/pkgs/sagelib/src to src
​a1a10b9	src/VERSION.txt: New
​5697335	src/setup.cfg: Add license_file=LICENSE.txt
​deb9eb3	Merge tag '9.3.beta3' into t/30912/sagelib__update_metadata_for_pypi_deployment
​7ad4c0e	Merge tag '9.3.beta4' into t/30912/sagelib__update_metadata_for_pypi_deployment
​27b589b	Merge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
​8796008	Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
​d6831b1	build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
​61f6ba6	Makefile: Add top-level targets build-local, build-venv
​2f6bb57	build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3c731b8

build/make/Makefile.in: Rework using tree_... variables

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d04b08b

build/make/Makefile.in: Define SAGE_VENV earlier

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​306ddef

WIP configure --with-sage-venv

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​bf6e5f8	build/make/Makefile.in: Change some SAGE_LOCAL to SAGE_VENV
​832f96a	build/make/Makefile.in: Rework using tree_... variables
​3436ad6	build/make/Makefile.in: Define SAGE_VENV earlier
​8cc0b3c	WIP configure --with-sage-venv

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1cfbe2b	build/bin/sage-spkg: Change more SAGE_LOCAL to SAGE_INST_LOCAL
​2202f84	configure.ac: Finish --with-sage-venv handling

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b604a68	build/make/Makefile.in: Also source sage-src-env-config for script packages
​92f1a1b	build/pkgs/pynac/spkg-install.in: Build with PYTHON_FOR_VENV

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1942363	build/pkgs/pynac/spkg-install.in: Back to PYTHON=python3
​9dcded5	src/bin/sage: Only do the check for an incomplete installation when run from the source tree

[jhpalmieri]

With ./configure --with-python=/usr/bin/python3 --with-sage-venv="`pwd`/local/var/lib/sage/venv/python3.8", I'm getting some doctest failures:

sage -t --long --warn-long 101.2 --random-seed=0 src/sage/misc/package.py  # 5 doctests failed

(because it doesn't find some Python packages). Also

sage -t --long --warn-long 101.2 --random-seed=0 src/sage_setup/find.py  # 2 doctests failed
sage -t --long --warn-long 101.2 --random-seed=0 src/sage_setup/clean.py  # 1 doctest failed

Details:

File "src/sage_setup/find.py", line 365, in sage_setup.find.installed_files_by_module
Failed example:
    f1
Expected:
    'sage/structure/__init__.py'
Got:
    '/Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.8/lib/python3.8/site-packages/sage/structure/__init__.cpython-38.pyc'
**********************************************************************
File "src/sage_setup/find.py", line 367, in sage_setup.find.installed_files_by_module
Failed example:
    f2
Expected:
    'sage/structure/....pyc'
Got:
    'sage/structure/__init__.py'
**********************************************************************
File "src/sage_setup/clean.py", line 96, in sage_setup.clean._find_stale_files
Failed example:
    for f in stale_iter:
        if f.endswith(skip_extensions): continue
        if '/ext_data/' in f: continue
        print('Found stale file: ' + f)
Expected nothing
Got:
    Found stale file: /Users/palmieri/Library/Caches/com.apple.python/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.8/lib/python3.8/site-packages/sage/all_cmdline.cpython-38.pyc
...
[followed by many many lines of "stale" files]

[mkoeppe]

The ones in find and clean are I think #31314.

For the failure in package I know what to do

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6e0c19e

sage.misc.package.installed_packages: After SAGE_SPKG_INST, also go through SAGE_VENV_SPKG_INST

[jhpalmieri]

The sage_setup doctest failures have vanished, but I am seeing different problems with misc/package.py:

**********************************************************************
File "src/sage/misc/package.py", line 554, in sage.misc.package.package_manifest
Failed example:
    sagetex_manifest = package_manifest('sagetex')  # optional - build
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.package_manifest[1]>", line 1, in <module>
        sagetex_manifest = package_manifest('sagetex')  # optional - build
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/misc/package.py", line 570, in package_manifest
        with open(stamp_file) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/installed/sagetex-3.5'
**********************************************************************
File "src/sage/misc/package.py", line 555, in sage.misc.package.package_manifest
Failed example:
    sagetex_manifest['package_name'] == 'sagetex'  # optional - build
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.package_manifest[2]>", line 1, in <module>
        sagetex_manifest['package_name'] == 'sagetex'  # optional - build
    NameError: name 'sagetex_manifest' is not defined
**********************************************************************
File "src/sage/misc/package.py", line 557, in sage.misc.package.package_manifest
Failed example:
    'files' in sagetex_manifest  # optional - build
Exception raised:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 714, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/palmieri/Desktop/Sage/sage_builds/TESTING/sage-9.3.rc1/local/var/lib/sage/venv/python3.9/lib/python3.9/site-packages/sage/doctest/forker.py", line 1133, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package.package_manifest[3]>", line 1, in <module>
        'files' in sagetex_manifest  # optional - build
    NameError: name 'sagetex_manifest' is not defined
**********************************************************************

I guess the issue is that manifests are now written to two different places, depending on whether it's a Python package or not.

[mkoeppe]

Right, I forgot about this function. It will need a slightly generalized interface

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c5dd7ba	Merge tag '9.3.rc2' into t/29013/public/29013-use-py3-venv
​f0fb082	sage.misc.package.installed_packages: Refactor through new generator _spkg_inst_dirs
​d68e861	sage.misc.package.package_manifest: Go through all _spkg_inst_dirs()

[jhpalmieri]

That fixes the package.py doctest for me, thank you. I'm not sure I fully understand how venvs are supposed to work. If I do ./configure --with-sage-venv=... && make and then separately do ./configure && make, I should get two builds (as far as Python packages are concerned), which I could conceivably switch between, right? Now if I do ./configure && make pysingular, then I can do import PySingular, but I switch to the other venv, I won't be able to. However, 'pysingular' in installed_packages() returns True regardless of whether I'm using the venv, since it searches both installation directories.

(a) Am I understanding the situation correctly? and

(b) is it worth trying to fix?

[mkoeppe]

Replying to jhpalmieri:

I'm not sure I fully understand how venvs are supposed to work. If I do ./configure --with-sage-venv=... && make and then separately do ./configure && make, I should get two builds (as far as Python packages are concerned), which I could conceivably switch between, right?

Yes, that's right. Separate for Python packages, sharing the non-Python packages, which are only in SAGE_LOCAL.

Now if I do ./configure && make pysingular, then I can do import PySingular, but I switch to the other venv, I won't be able to. However, 'pysingular' in installed_packages() returns True regardless of whether I'm using the venv, since it searches both installation directories.

OK, thanks for catching this. Yes, the sage.misc.package code does not handle this situation correctly. I would simply suggest to not mix builds that use with --with-sage-venv=... with builds that don't. You can switch between several venvs in this way.

[jhpalmieri]

Great, let's merge it, then.

[mkoeppe]

Thank you!