#29950 closed enhancement (fixed)

Build sagelib from build/pkgs/sagelib/src, fix `setup.py sdist`, add spkg-src and tox.ini

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.2
Component: refactoring Keywords:
Cc: fbissey, jhpalmieri, dimpase, gh-kliem, isuruf, vbraun Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: ccc67b0 (Commits, GitHub, GitLab) Commit: ccc67b0356373330164dd75060f574c184142473
Dependencies: #29701 Stopgaps:

Status badges

Description (last modified by mkoeppe)

As a follow-up to #29411, we set up a separate directory build/pkgs/sagelib/src with symlinks into SAGE_ROOT/src so that sagelib's setup.py does not have access to the whole tree SAGE_ROOT/src.

We fix setup.py sdist, which can be tested using spkg-src (which builds a pip-installable tarball in upstream/) and tox:

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -v -v)'
./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage-ipython'

The call of ./sage -sh is necessary so that SAGE_LOCAL is set. Moreover, for scripts that use sage-env, also SAGE_ENV_SOURCED is crucial for the correct operation; otherwise sage-env sets things based on SAGE_LOCAL again.

Change History (44)

comment:1 Changed 16 months ago by mkoeppe

  • Branch set to u/mkoeppe/build_sagelib_using_installed_sage_setup

comment:2 Changed 16 months ago by git

  • Commit set to 1ab75a99c46a00e7424a5d91366097fc32c9a13a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

0459006.gitignore: Ignore generated files in build/pkgs/*/src
1ab75a9Build sagelib using the installed sage_setup

comment:3 Changed 16 months ago by mkoeppe

  • Summary changed from Build sagelib using the installed sage_setup to Build sagelib using the installed sage_setup, add spkg-src

comment:4 Changed 16 months ago by mkoeppe

  • Work issues set to add spkg-src

comment:5 Changed 16 months ago by mkoeppe

  • Authors set to Matthias Koeppe

comment:6 Changed 16 months ago by git

  • Commit changed from 1ab75a99c46a00e7424a5d91366097fc32c9a13a to a29cd2d79dd84d9653f9f0ea38466f846edb0e4f

Branch pushed to git repo; I updated commit sha1. New commits:

3d83c12build/pkgs/sagelib/spkg-src: New
a2bde50src/MANIFEST.in: prune .tox
4b258d3src/MANIFEST.in: Remove bitrotted file
a29cd2dbuild/pkgs/sagelib/MANIFEST.in: New

comment:7 Changed 16 months ago by mkoeppe

  • Work issues changed from add spkg-src to Fixup MANIFEST.in

comment:8 Changed 16 months ago by mkoeppe

sdist is broken: Files named ** end up in the tar file.

comment:9 Changed 16 months ago by mkoeppe

  • Dependencies changed from #29411, #29847 to #29411, #29847, #29701

comment:11 Changed 16 months ago by git

  • Commit changed from a29cd2d79dd84d9653f9f0ea38466f846edb0e4f to c3b8f8f057647eca8725a7a35a415c1cc3e19d4b

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

7f8850asrc/module_list.py: Document that it is obsolete
26a85a4Merge branch 't/29702/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' into t/29701/replace_use_of_module_list_optionalextension
5d5803esrc/sage/graphs/planarity.pyx: Add forgotten distutils directive
a0be9b6Merge branch 't/29790/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_5__sage_graphs_' into t/29701/replace_use_of_module_list_optionalextension
14396afMerge tag '9.2.beta2' into t/29701/replace_use_of_module_list_optionalextension
821e337Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup
eeb6ba4build/pkgs/sagelib/src/requirements.txt, tox.ini: New
ce8b84dbuild/pkgs/sagelib/src/MANIFEST.in: Prune sage/ext/interpreters
a9a1335build/pkgs/sagelib/src/requirements.txt: Add packages required for runtime
c3b8f8fSuggest tox -v -v

comment:12 Changed 16 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Build sagelib using the installed sage_setup, add spkg-src to Build sagelib from build/pkgs/sagelib/src, fix `setup.py sdist`, add spkg-src and tox.ini
  • Work issues changed from Fixup MANIFEST.in to Rebase away from #29847

comment:13 Changed 16 months ago by git

  • Commit changed from c3b8f8f057647eca8725a7a35a415c1cc3e19d4b to 9955a3ff69870ff844e5ebb45cf47c4b6f214263

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

cdf501fbuild/pkgs/sagelib/spkg-src: New
ef4f5b8Move src/MANIFEST.in to build/pkgs/sagelib/MANIFEST.in, fix up
233f6babuild/pkgs/sagelib/src/requirements.txt, tox.ini: New
177cd4cbuild/pkgs/sagelib/src/MANIFEST.in: Prune sage/ext/interpreters
07b29fbbuild/pkgs/sagelib/src/requirements.txt: Add packages required for runtime
ad33705Suggest tox -v -v
481e212.gitignore: Ignore generated files in build/pkgs/*/src
9955a3fBuild sagelib from build/pkgs/sagelib/src

comment:14 Changed 16 months ago by mkoeppe

  • Dependencies changed from #29411, #29847, #29701 to #29411, #29701
  • Work issues Rebase away from #29847 deleted

comment:15 Changed 16 months ago by mkoeppe

    from .maxima import maxima, Maxima
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/lib/python3.7/site-packages/sage/interfaces/maxima.py", line 1236, in <module>
    script_subdirectory=None)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/lib/python3.7/site-packages/sage/interfaces/maxima.py", line 528, in __init__
    raise RuntimeError('You must get the file local/bin/sage-maxima.lisp')
RuntimeError: You must get the file local/bin/sage-maxima.lisp

comment:16 Changed 16 months ago by mkoeppe

  • Dependencies changed from #29411, #29701 to #29411, #29701, #27171

comment:17 Changed 16 months ago by mkoeppe

  • Dependencies changed from #29411, #29701, #27171 to #29411, #29701, #21559

comment:18 Changed 16 months ago by git

  • Commit changed from 9955a3ff69870ff844e5ebb45cf47c4b6f214263 to 08f5175bdf71236ccf3e80f6efaffa83d5edc532

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

875940dtrac 29111: more tinkering. Fix doctests in tests/cmdline.py.
4a3d36eMove 'sage -app' back to src/bin/sage
3a0193csrc/bin/sage: Remove handling of 'sage -axiom'
6b04075Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation
9c7116bsrc/bin/sage-list-optional, sage-list-experimental, sage-list-standard: Remove deprecated scripts
831cc09Merge branch 't/29920/remove_deprecated_scripts_sage_list_optional__sage_list_experimental__sage_list_standard' into t/21559/change-src-bin-installation
a56dc35Merge tag '9.2.beta1' into t/29702/public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup
e3eca85Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/21559/change-src-bin-installation
7d29141src/setup.py: Do not install removed script sage-rsyncdist
08f5175Merge branch 'u/mkoeppe/change-src-bin-installation' of git://trac.sagemath.org/sage into t/29950/build_sagelib_using_installed_sage_setup

comment:19 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:20 Changed 16 months ago by mkoeppe

./sage -sh -c '(cd build/pkgs/sagelib/src && tox -r -v -v)'
...
RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work
ERROR: InvocationError for command /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python/bin/python -c 'import sys; "" in sys.path and sys.path.remove(""); import sage.all' (exited with code 1)

This is because of:

$ build/pkgs/sagelib/src/.tox/python/bin/python
Python 3.7.7 (v3.7.7:d7c567b08f, Mar 10 2020, 02:56:16) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sage.env
>>> sage.env.SINGULAR_SO
>>> sage.env.GAP_SO
>>> sage.env.SAGE_LOCAL
'/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/build/pkgs/sagelib/src/.tox/python'
>>> 

comment:21 Changed 16 months ago by mkoeppe

Passing the correct SAGE_LOCAL suffices to fix this, of course. But the singular library location should be configured separately (#29024).

comment:22 Changed 16 months ago by git

  • Commit changed from 08f5175bdf71236ccf3e80f6efaffa83d5edc532 to 516358b66059dbf458ed7c970ff05923b539b71c

Branch pushed to git repo; I updated commit sha1. New commits:

516358bbuild/pkgs/sagelib/src/tox.ini: Pass SAGE_LOCAL

comment:23 Changed 16 months ago by git

  • Commit changed from 516358b66059dbf458ed7c970ff05923b539b71c to eaced098339d5c7655244667b3dd1c90f5fd7f9e

Branch pushed to git repo; I updated commit sha1. New commits:

d62da15sage_setup.command.sage_build: Add the extensions to the distribution
5e577abMerge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup
eaced09build/pkgs/sagelib: Add remaining Python dependencies for sage.all

comment:24 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:25 Changed 16 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from new to needs_review

comment:26 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:27 Changed 16 months ago by mkoeppe

The sage script relies on PATH to be set correctly when invoking auxiliary scripts such as sage-eval. Hence the following does not work correctly:

$ ./sage -sh -c 'build/pkgs/sagelib/src/.tox/python/bin/sage -c "print(sys.path)"'
['/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/bin', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python37.zip', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7', '/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload', '/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages']

comment:28 Changed 16 months ago by mkoeppe

This will be fixed by #30013 (src/bin/sage-env: Make sure SAGE_SCRIPTS_DIR is at the beginning of the PATH).

comment:29 Changed 16 months ago by git

  • Commit changed from eaced098339d5c7655244667b3dd1c90f5fd7f9e to b8e366ec56a9d3527c81df9d8d4206afc2562be4

Branch pushed to git repo; I updated commit sha1. New commits:

b8e366ebuild/pkgs/sagelib/spkg-src: chmod +x

comment:30 Changed 16 months ago by mkoeppe

  • Cc jhpalmieri dimpase gh-kliem isuruf added

comment:31 Changed 16 months ago by mkoeppe

All dependencies of this ticket are green. Ready for review.

comment:32 Changed 16 months ago by git

  • Commit changed from b8e366ec56a9d3527c81df9d8d4206afc2562be4 to 72443711047daa51d1ba92b087ec9022b2eeef06

Branch pushed to git repo; I updated commit sha1. New commits:

c5f7f2fMerge tag '9.2.beta3' into t/21559/change-src-bin-installation
0473ef3Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup
06a3609sage_setup.find.find_python_sources: Add benchmark doctest
174626cMerge branch 't/29786/move_extension_options_from_src_module_list_py_to__distutils___directives_in_the_individual_files__part_4__sage_rings_' into t/29701/replace_use_of_module_list_optionalextension
034a7f7Merge branch 'public/move_all_code_from_src_setup_py__src_fpickle_setup_py_to_sage_setup' of git://trac.sagemath.org/sage into t/29701/replace_use_of_module_list_optionalextension
d3c608bMerge tag '9.2.beta3' into t/29701/replace_use_of_module_list_optionalextension
7244371Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup

comment:33 Changed 16 months ago by mkoeppe

Merged in latest versions #29701, #21559 (on top of 9.2.beta3). Needs review

comment:34 Changed 16 months ago by git

  • Commit changed from 72443711047daa51d1ba92b087ec9022b2eeef06 to 4344f8940fc44d959da5d20dbea43b64181b961f

Branch pushed to git repo; I updated commit sha1. New commits:

3953671Merge tag '9.2.beta4' into t/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution
19d7f67Merge branch 't/29111/specify_a_subset_of_sage_command_line_options_that_are_supported_by_sagelib___rather_than_sage_the_distribution' into t/21559/change-src-bin-installation
4344f89Merge branch 't/21559/change-src-bin-installation' into t/29950/build_sagelib_using_installed_sage_setup

comment:35 Changed 15 months ago by git

  • Commit changed from 4344f8940fc44d959da5d20dbea43b64181b961f to 01b96b0f44f4ff7407fd2f3ce83de1cc74259d58

Branch pushed to git repo; I updated commit sha1. New commits:

fcad518Merge tag '9.2.beta7' into t/29701/replace_use_of_module_list_optionalextension
55c3fbcsrc/sage_setup/clean.py: Fix doctest
01b96b0Merge branch 't/29701/replace_use_of_module_list_optionalextension' into t/29950/build_sagelib_using_installed_sage_setup

comment:36 Changed 15 months ago by mkoeppe

  • Dependencies changed from #29411, #29701, #21559 to #29701

comment:37 Changed 15 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

lgtm

comment:38 Changed 15 months ago by mkoeppe

  • Cc vbraun added
  • Status changed from positive_review to needs_work

Failing doctest needs work as noted in #29701.

comment:39 Changed 15 months ago by git

  • Commit changed from 01b96b0f44f4ff7407fd2f3ce83de1cc74259d58 to 8a19fe2a9bf4ed64a48ac3110cccbbdff97684ca

Branch pushed to git repo; I updated commit sha1. New commits:

8a19fe2build/make/Makefile.in (sagelib-clean): Clean the new build location

comment:40 Changed 15 months ago by git

  • Commit changed from 8a19fe2a9bf4ed64a48ac3110cccbbdff97684ca to ccc67b0356373330164dd75060f574c184142473

Branch pushed to git repo; I updated commit sha1. New commits:

ccc67b0src/sage_setup: Update cythonized_dir in doctests

comment:41 Changed 15 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:42 Changed 15 months ago by dimpase

  • Status changed from needs_review to positive_review

ok

comment:43 Changed 15 months ago by mkoeppe

Thanks!

comment:44 Changed 15 months ago by vbraun

  • Branch changed from u/mkoeppe/build_sagelib_using_installed_sage_setup to ccc67b0356373330164dd75060f574c184142473
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.