Opened 21 months ago

Last modified 10 days ago

#29039 needs_review enhancement

pip-installable version of package sage_conf - installs non-Python bits of the Sage distribution in ~/.sage/

Reported by: mkoeppe Owned by:
Priority: critical Milestone: sage-9.5
Component: build Keywords: sd111
Cc: jhpalmieri, dimpase, fbissey Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik, ...
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/pip_installable_package_sage_bootstrap (Commits, GitHub, GitLab) Commit: c695921c3cee61a32a8d7ec9443bb4e315eb481b
Dependencies: #32442 Stopgaps:

Status badges

Description (last modified by mkoeppe)

As #30913 makes sage-conf both (a) a PEP517/518 build_system requires and (b) an install-requires of sagelib (sagemath-standard), also a version of sage-conf must be made available on PyPI.

This version of sage-conf includes a copy of SAGE_ROOT and SAGE_ROOT/build/** in its source distribution. On installation (or wheel building) it establishes a python-less SAGE_LOCAL hierarchy (#31362) in ~/.sage/ by running configure and make build-local; and then installs the configured sage_conf.py (#29038) as a Python module (and sage-config, sage-env-config).

There are no changes to how the Sage distribution is built. This ticket only provides an alternative and strictly optional way of installing (the non-Python bits of) Sage-the-distribution. It may be useful for some users who wish to use pip to install everything they need.

To test:

./bootstrap && (cd pkgs/sage-conf_pypi/ && tox -r -v -v -v)

This test using tox uses .sage in an isolated directory within .tox instead of in your home directory. At the end of the tox run, the installed sage-config script is executed; it prints the configuration variables:

[10639] /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/pkgs/sage-conf_pypi$ /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/pkgs/sage-conf-pypi/.tox/python/bin/sage-config
VERSION=9.3.beta7
MAXIMA=/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.beta7-cpython-39-darwin/local/bin/maxima
ARB_LIBRARY=arb
SAGE_NAUTY_BINS_PREFIX=
CBLAS_PC_MODULES=cblas
MATHJAX_DIR=/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.beta7-cpython-39-darwin/local/share/mathjax
THREEJS_DIR=/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.beta7-cpython-39-darwin/local/share/threejs
SAGE_LOCAL=/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.beta7-cpython-39-darwin/local
SAGE_ROOT=/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.beta7-cpython-39-darwin
python finish: run-test  after 0.17 seconds

To build a source distribution:

./sage -sh -c build/pkgs/sage_conf/spkg-src

(The source distribution has been uploaded to https://pypi.org/project/sage-conf/)

Follow-up steps:

  • Make SAGE_ROOT configurable by the user (by passing --prefix to setup.py).
  • Optionally, it would create a venv that is used for building wheels.
  • #30036: Make the build more incremental: On installation of sage-conf, only run configure but do not install anything; rather, a Python entry point can be used by Python distribution packages such as sagemath-standard to install packages if present.

Change History (153)

comment:1 Changed 21 months ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-9.1 to sage-wishlist

comment:2 Changed 16 months ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-wishlist to sage-9.3
  • Summary changed from Package sage_local to pip-installable package sage_bootstrap

comment:3 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 16 months ago by mkoeppe

  • Dependencies set to #30025
  • Description modified (diff)

comment:5 Changed 16 months ago by mkoeppe

  • Branch set to u/mkoeppe/pip_installable_package_sage_bootstrap

comment:6 Changed 16 months ago by mkoeppe

  • Commit set to 6e28ae56c3eab514bd350182640e00489dc81ab0
  • Description modified (diff)

New commits:

6e28ae5Add sage_root to sage_bootstrap

comment:7 Changed 16 months ago by git

  • Commit changed from 6e28ae56c3eab514bd350182640e00489dc81ab0 to a6373b762d6a35cf51e48a005d9a5b04bcc621f9

Branch pushed to git repo; I updated commit sha1. New commits:

a6373b7build/setup.py: Add sage_root as package_data

comment:8 Changed 16 months ago by git

  • Commit changed from a6373b762d6a35cf51e48a005d9a5b04bcc621f9 to b0ddab043224bf868bc0753ed2afbf1e3ad9404c

Branch pushed to git repo; I updated commit sha1. New commits:

441cb6eWorking sdist and install
b0ddab0Simplify directory structure

comment:9 Changed 16 months ago by mkoeppe

  • Dependencies changed from #30025 to #30025, #29850

comment:10 Changed 16 months ago by git

  • Commit changed from b0ddab043224bf868bc0753ed2afbf1e3ad9404c to 60642d345c9961990f0e126be9af58ab3f893143

Branch pushed to git repo; I updated commit sha1. New commits:

10f4fbebuild/setup.py: Reuse configured sage root
733427cRe-add build/sage_root
60642d3build/setup.py: Install sage_conf.py, sage-env-config

comment:11 Changed 16 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 16 months ago by mkoeppe

  • Dependencies changed from #30025, #29850 to #30025

comment:13 Changed 16 months ago by git

  • Commit changed from 60642d345c9961990f0e126be9af58ab3f893143 to 8006b6f9d6a2b531f067b42d416f00d591d63aaf

Branch pushed to git repo; I updated commit sha1. New commits:

8006b6fAdd sage_bootstrap.build

comment:14 Changed 16 months ago by mkoeppe

  • Work issues set to sage-system-python in installed sage_bootstrap

comment:15 Changed 15 months ago by mkoeppe

  • Authors set to Matthias Koeppe

comment:16 Changed 13 months ago by mkoeppe

  • Dependencies changed from #30025 to #30025, #30527
  • Description modified (diff)

comment:17 Changed 13 months ago by mkoeppe

  • Dependencies changed from #30025, #30527 to #30025, #30527, #29013
  • Description modified (diff)

comment:18 Changed 12 months ago by mkoeppe

  • Dependencies changed from #30025, #30527, #29013 to #30527, #29013
  • Description modified (diff)

comment:19 Changed 11 months ago by mkoeppe

  • Dependencies #30527, #29013 deleted

comment:20 Changed 11 months ago by git

  • Commit changed from 8006b6f9d6a2b531f067b42d416f00d591d63aaf to 9293ae1d878c7fa24a3098687c49757027be361b

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

a1ad0beAdd sage_root to sage_bootstrap
8695368build/setup.py: Add sage_root as package_data
cc93ecaWorking sdist and install
45c0ebeSimplify directory structure
62d4f6cbuild/setup.py: Reuse configured sage root
7d705e3Re-add build/sage_root
44e3224build/setup.py: Install sage_conf.py, sage-env-config
9293ae1Add sage_bootstrap.build

comment:21 Changed 11 months ago by git

  • Commit changed from 9293ae1d878c7fa24a3098687c49757027be361b to 3f243e19bd975f4c71b34a049f90d0ff2097a8d0

Branch pushed to git repo; I updated commit sha1. New commits:

05c44c8build/sage_root: Remove outdated symlink
3f243e1build/setup.py: Use configure --with-python=...

comment:22 Changed 11 months ago by mkoeppe

  • Keywords sd111 added

comment:23 Changed 9 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from pip-installable package sage_bootstrap to pip-installable packages sage_bootstrap, sage_conf

comment:24 Changed 9 months ago by mkoeppe

  • Dependencies set to #30913

comment:25 Changed 9 months ago by mkoeppe

  • Work issues changed from sage-system-python in installed sage_bootstrap to sage-system-python in installed sage_bootstrap. Rework it so that sage_conf is the package running configure.

comment:26 Changed 8 months ago by git

  • Commit changed from 3f243e19bd975f4c71b34a049f90d0ff2097a8d0 to b7cd894f49dd4f39824f421166faf281fb64d641

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

7da13f4Add sage_root to sage_bootstrap
9ea9d80build/setup.py: Add sage_root as package_data
3f5c99cWorking sdist and install
1f7647fSimplify directory structure
f59d077build/setup.py: Reuse configured sage root
4a18b98Re-add build/sage_root
8e44d28build/setup.py: Install sage_conf.py, sage-env-config
86e21a1Add sage_bootstrap.build
47f5e95build/sage_root: Remove outdated symlink
b7cd894build/setup.py: Use configure --with-python=...

comment:27 Changed 8 months ago by git

  • Commit changed from b7cd894f49dd4f39824f421166faf281fb64d641 to 44b873b3023f487e0188554b1d353520db41f884

Branch pushed to git repo; I updated commit sha1. New commits:

4ff9371Move installation of SAGE_ROOT from sage_bootstrap to sage_conf (pypi version)
44b873bPut both versions of sage_conf sources into subdirectories of SAGE_ROOT/src/pkgs/

comment:28 Changed 8 months ago by mkoeppe

  • Dependencies changed from #30913 to #30913, #31362
  • Description modified (diff)
  • Summary changed from pip-installable packages sage_bootstrap, sage_conf to pip-installable version of package sage_conf

comment:29 Changed 8 months ago by git

  • Commit changed from 44b873b3023f487e0188554b1d353520db41f884 to 8ee90871c44a31ebf3a48932567475ae1bcec9b5

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8ee9087Put both versions of sage_conf sources into subdirectories of SAGE_ROOT/src/pkgs/

comment:30 Changed 8 months ago by git

  • Commit changed from 8ee90871c44a31ebf3a48932567475ae1bcec9b5 to 47e182d81ec5fa6fa810fd30316bdf42047b067c

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

47e182dPut both versions of sage_conf sources into subdirectories of SAGE_ROOT/src/pkgs/

comment:31 Changed 8 months ago by mkoeppe

  • Dependencies changed from #30913, #31362 to #30913, #31362, #31357

comment:32 Changed 8 months ago by git

  • Commit changed from 47e182d81ec5fa6fa810fd30316bdf42047b067c to b2b785b2b9ae6194a7d15f2f1b1ac694ed388e84

Branch pushed to git repo; I updated commit sha1. New commits:

55bc2aam4/sage_spkg_collect.m4: Generate install tree information for build/make/Makefile
d7dc9a2build/make/Makefile.in: New targets all-sage-local, all-sage-venv
27b589bMerge branch 't/30912/sagelib__update_metadata_for_pypi_deployment' into t/31362/make_all_sage_local__make_all_sage_venv
8796008Put pynac into SAGE_LOCAL, sagelib into SAGE_VENV
d6831b1build/make/Makefile.in: Add all-build-local, all-build-venv, which include dependency on toolchain
61f6ba6Makefile: Add top-level targets build-local, build-venv
b2b785bPut both versions of sage_conf sources into subdirectories of SAGE_ROOT/src/pkgs/

comment:33 Changed 8 months ago by mkoeppe

  • Description modified (diff)
  • Work issues sage-system-python in installed sage_bootstrap. Rework it so that sage_conf is the package running configure. deleted

comment:34 Changed 8 months ago by mkoeppe

  • Dependencies changed from #30913, #31362, #31357 to #30913, #31362, #31357, #31321
  • Description modified (diff)

comment:35 Changed 8 months ago by git

  • Commit changed from b2b785b2b9ae6194a7d15f2f1b1ac694ed388e84 to b8b8a0607d3a9d7ad05df8c8e306c41c82a05e24

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b8b8a06Put both versions of sage_conf sources into subdirectories of SAGE_ROOT/src/pkgs/

comment:36 Changed 8 months ago by git

  • Commit changed from b8b8a0607d3a9d7ad05df8c8e306c41c82a05e24 to ff7f110f406e3a88ae655b7c77b8f0f93d213006

Branch pushed to git repo; I updated commit sha1. New commits:

ff7f110build/pkgs/sage_conf/install-requires.txt: New

comment:37 Changed 8 months ago by git

  • Commit changed from ff7f110f406e3a88ae655b7c77b8f0f93d213006 to 072b522e2e10bdcec729f72dcaa82908154864b7

Branch pushed to git repo; I updated commit sha1. New commits:

072b522Determine SAGE_ROOT from sage version and python version

comment:38 Changed 8 months ago by git

  • Commit changed from 072b522e2e10bdcec729f72dcaa82908154864b7 to fa4cbfa8bf933c41c1a6d0d649fb0fb3b5030aa1

Branch pushed to git repo; I updated commit sha1. New commits:

fa4cbfaRun .homebrew-build-env and set 'make -j...' automatically

comment:39 Changed 8 months ago by mkoeppe

  • Description modified (diff)
  • Priority changed from major to critical
  • Status changed from new to needs_review
  • Summary changed from pip-installable version of package sage_conf to pip-installable version of package sage_conf - installs non-Python bits of the Sage distribution in ~/.sage/

comment:40 Changed 8 months ago by mkoeppe

  • Cc jhpalmieri dimpase fbissey added
  • Description modified (diff)

comment:41 Changed 8 months ago by git

  • Commit changed from fa4cbfa8bf933c41c1a6d0d649fb0fb3b5030aa1 to fa9485cfc9a47aec79c77f73575c56c56c197666

Branch pushed to git repo; I updated commit sha1. New commits:

fa9485cRemove sage_bootstrap.build (not needed for this ticket)

comment:42 Changed 8 months ago by mkoeppe

  • Description modified (diff)

comment:43 Changed 8 months ago by git

  • Commit changed from fa9485cfc9a47aec79c77f73575c56c56c197666 to 7bdc52452f725002185d58d95735f7ee191ce469

Branch pushed to git repo; I updated commit sha1. New commits:

7bdc524build/pkgs/sage_conf/src: Restore symlink

comment:44 Changed 8 months ago by git

  • Commit changed from 7bdc52452f725002185d58d95735f7ee191ce469 to 76d151e35870c8e52a2423058c497e3e6252b517

Branch pushed to git repo; I updated commit sha1. New commits:

76d151ebuild/bin/write-dockerfile.sh: Fix up docker tests

comment:45 Changed 8 months ago by git

  • Commit changed from 76d151e35870c8e52a2423058c497e3e6252b517 to 1a8cadc8975ce9d7d662678c7a8019202c084b46

Branch pushed to git repo; I updated commit sha1. New commits:

1a8cadcsrc/pkgs/sage_conf-pypi/setup.cfg: Under version control, not gitignored

comment:46 Changed 8 months ago by git

  • Commit changed from 1a8cadc8975ce9d7d662678c7a8019202c084b46 to 31cadddccf1901b71c194aa1617929b738cf131d

Branch pushed to git repo; I updated commit sha1. New commits:

31cadddMerge tag '9.3.beta8' into t/29039/pip_installable_package_sage_bootstrap

comment:47 Changed 7 months ago by git

  • Commit changed from 31cadddccf1901b71c194aa1617929b738cf131d to 48d2a2bf872e15804f7125e39dc3abb19d1eaf49

Branch pushed to git repo; I updated commit sha1. New commits:

f691585Merge tag '9.3.beta9' into t/30913/sagelib__add_setup_cfg__install_requires_
48d2a2bMerge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap

comment:48 Changed 7 months ago by mkoeppe

  • Dependencies changed from #30913, #31362, #31357, #31321 to #30913

comment:49 Changed 7 months ago by mkoeppe

  • Dependencies changed from #30913 to #30913, #30383

comment:50 Changed 7 months ago by git

  • Commit changed from 48d2a2bf872e15804f7125e39dc3abb19d1eaf49 to 5dbd712fa100252e2472a4dd33095bbe1a4c5a7a

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

82d8030m4/sage_spkg_enable.m4: Cosmetic change to help strings
3e92998configure --disable-notebook: Fix up what packages are disabled
74a83fcbuild/pkgs/matplotlib/dependencies: Undo removal of tornado
93d31b9Remove use of unicode superscripts in configure --help
18fbb85m4/sage_spkg_collect.m4: Fix description of SAGE_OPTIONAL_CLEANED_PACKAGES
c3e4093Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES
4916415Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default
a35428eMerge #30383
e098133src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS
5dbd712src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built

comment:51 Changed 7 months ago by git

  • Commit changed from 5dbd712fa100252e2472a4dd33095bbe1a4c5a7a to 100ff515319f6f7fbe66487a5bb35fc34cd3b07f

Branch pushed to git repo; I updated commit sha1. New commits:

839c534src/pkgs/sage_conf-pypi/setup.py: Fix typo
100ff51src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src

comment:52 Changed 7 months ago by git

  • Commit changed from 100ff515319f6f7fbe66487a5bb35fc34cd3b07f to a4f75b86bf2c70bbc4edb7f0d5ee558ccf75e1b0

Branch pushed to git repo; I updated commit sha1. New commits:

afa2ffdAdd configure option --disable-sagelib
a4f75b8src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README

comment:53 Changed 7 months ago by git

  • Commit changed from a4f75b86bf2c70bbc4edb7f0d5ee558ccf75e1b0 to ac0f563ba1fff7ed111499bead102573d83dc1c8

Branch pushed to git repo; I updated commit sha1. New commits:

66c2987build/pkgs/{sage_sws2rst,sage_docbuild}/install-requires.txt: New
7f8ec6cbuild/pkgs/sagelib/src/tox.ini: Add factor sitepackages
0283da5build/make/Makefile.in: Add wheel, setuptools_wheel to PYTHON_TOOLCHAIN to make sure that PEP 517 packages have a complete build system
f720722build/pkgs/sagelib/src/tox.ini: Add factor nobuildisolation
c451b31src/setup.cfg.m4 (install_requires): Add sage_conf
6700223Merge tag '9.3.rc0' into t/30913/sagelib__add_setup_cfg__install_requires_
ac0f563Merge branch 't/30913/sagelib__add_setup_cfg__install_requires_' into t/29039/pip_installable_package_sage_bootstrap

comment:54 Changed 7 months ago by mkoeppe

  • Dependencies changed from #30913, #30383 to #30913

comment:55 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:56 Changed 7 months ago by mkoeppe

  • Description modified (diff)

comment:57 follow-up: Changed 7 months ago by dimpase

what does the change to build/pkgs/sagelib/src/MANIFEST.in mean? is it becoming a symbolic link?

comment:58 follow-up: Changed 7 months ago by dimpase

after merging 9.3.rc2 in, I get an error (is it from sh: line 0: .: .homebrew-build-env: file not found, cf below ?):

$ ./bootstrap && (cd src/pkgs/sage_conf-pypi/ && tox -v -v -v)
...
    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    writing manifest file '/home/scratch2/dimpase/tmp/pip-modern-metadata-27nf1gz1/sage_conf.egg-info/SOURCES.txt'
    creating '/home/scratch2/dimpase/tmp/pip-modern-metadata-27nf1gz1/sage_conf.dist-info'
    Preparing wheel metadata ... done
  Source in /home/scratch2/dimpase/tmp/pip-req-build-0rrkid61 has version 9.3rc2, which satisfies requirement sage-conf==9.3rc2 from file:///home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3rc2.zip
  Removed sage-conf==9.3rc2 from file:///home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3rc2.zip from build tracker '/home/scratch2/dimpase/tmp/pip-req-tracker-bla_u_md'
Created temporary directory: /home/scratch2/dimpase/tmp/pip-unpack-rwlm13mn
Building wheels for collected packages: sage-conf
  Created temporary directory: /home/scratch2/dimpase/tmp/pip-wheel-k4duer3n
  Destination directory: /home/scratch2/dimpase/tmp/pip-wheel-k4duer3n
  Running command /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /home/scratch2/dimpase/tmp/tmpdhib0roj
  running bdist_wheel
  running build
  running build_py
  sh: line 0: .: .homebrew-build-env: file not found
  Running cd /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu && . .homebrew-build-env 2>&1; : && ./configure --prefix=/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu/local --with-python=/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python --with-system-python3=force --disable-notebook --disable-sagelib
  error: configure failed
  Building wheel for sage-conf (PEP 517) ... error
  ERROR: Failed building wheel for sage-conf
Failed to build sage-conf
ERROR: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 361, in run
    ", ".join(pep517_build_failure_names)
pip._internal.exceptions.InstallationError: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Removed build tracker: '/home/scratch2/dimpase/tmp/pip-req-tracker-bla_u_md'
python finish: installpkg after 3.39 seconds
python start: envreport 
setting PATH=/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin:/users/dimpase/perl5/bin:/users/dimpase/bin:/usr/libexec/python3-sphinx:/usr/lib64/qt-3.3/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/users/dimpase/.cabal/bin:/auto/users/dimpase/bin
  /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi$ /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip freeze >/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/log/python-3.log
python finish: envreport after 0.20 seconds
python installed: 
_________________________________________________________________________________________________ summary __________________________________________________________________________________________________
ERROR:   python: InvocationError for command '/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --no-deps -U -v /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3rc2.zip' (exited with code 1)
cleanup /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3rc2.zip

comment:59 Changed 7 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to needs_work

this is on Fedora 30, in case it matters

comment:60 in reply to: ↑ 57 Changed 6 months ago by mkoeppe

Replying to dimpase:

what does the change to build/pkgs/sagelib/src/MANIFEST.in mean? is it becoming a symbolic link?

Yes, it was duplicated with SAGE_ROOT/src/MANIFEST.in, and I replaced it with a symlink

comment:61 Changed 6 months ago by git

  • Commit changed from ac0f563ba1fff7ed111499bead102573d83dc1c8 to 1b03fa9a4f14439d0dbc3bd1af194ab25fa9eeb8

Branch pushed to git repo; I updated commit sha1. New commits:

9a3aa38build/bin/write-dockerfile.sh: ADD some missing files in SAGE_ROOT
1b03fa9src/pkgs/sage_conf-pypi/setup.py: Change SETENV so that it works in shells with set -e

comment:62 in reply to: ↑ 58 Changed 6 months ago by mkoeppe

Replying to dimpase:

after merging 9.3.rc2 in, I get an error (is it from sh: line 0: .: .homebrew-build-env: file not found, cf below ?):

$ ./bootstrap && (cd src/pkgs/sage_conf-pypi/ && tox -v -v -v)
...

Thanks for testing! Is this running in a docker container? Not clear to me why the file would be missing. Anyway I have cherry-picked a related change from #31396 and made another change - please take a look if this gets farther

comment:63 Changed 6 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:64 Changed 6 months ago by dimpase

no, still the same error:

...
Created temporary directory: /home/scratch2/dimpase/tmp/pip-unpack-169y8koj
Building wheels for collected packages: sage-conf
  Created temporary directory: /home/scratch2/dimpase/tmp/pip-wheel-exb6ynmx
  Destination directory: /home/scratch2/dimpase/tmp/pip-wheel-exb6ynmx
  Running command /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /home/scratch2/dimpase/tmp/tmpvehv3h0s
  running bdist_wheel
  running build
  running build_py
  sh: line 0: .: .homebrew-build-env: file not found
  Running cd /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu && (. .homebrew-build-env 2>&1 || :) && ./configure --prefix=/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu/local --with-python=/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python --with-system-python3=force --disable-notebook --disable-sagelib
  error: configure failed
  Building wheel for sage-conf (PEP 517) ... error
  ERROR: Failed building wheel for sage-conf
Failed to build sage-conf
ERROR: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 361, in run
    ", ".join(pep517_build_failure_names)
pip._internal.exceptions.InstallationError: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Removed build tracker: '/home/scratch2/dimpase/tmp/pip-req-tracker-_6q2pk49'
python finish: installpkg after 3.35 seconds
python start: envreport 
setting PATH=/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin:/users/dimpase/perl5/bin:/users/dimpase/bin:/usr/libexec/python3-sphinx:/usr/lib64/qt-3.3/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/users/dimpase/.cabal/bin:/auto/users/dimpase/bin
  /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi$ /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip freeze >/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/log/python-9.log
python finish: envreport after 0.20 seconds
python installed: 
__________________________________________________________________________ summary ___________________________________________________________________________
ERROR:   python: InvocationError for command '/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --no-deps -U -v /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3rc2.zip' (exited with code 1)

comment:65 Changed 6 months ago by dimpase

  • Status changed from needs_review to needs_work

comment:66 Changed 6 months ago by mkoeppe

Could you show what's in /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu?

comment:67 Changed 6 months ago by dimpase

$ ls -la /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu
total 1196
drwxr-xr-x 6 dimpase staff    4096 Apr  9 20:24 .
drwxr-xr-x 4 dimpase staff    4096 Apr  9 20:24 ..
-rwxr-xr-x 1 dimpase staff      81 Apr  9 20:24 bootstrap
drwxr-xr-x 7 dimpase staff    4096 Apr  9 20:24 build
drwxr-xr-x 2 dimpase staff    4096 Apr  9 20:24 config
-rwxr-xr-x 1 dimpase staff 1126829 Apr  9 20:24 configure
-rw-r--r-- 1 dimpase staff   18020 Apr  9 20:24 configure.ac
-rw-r--r-- 1 dimpase staff    1602 Apr  9 20:24 .homebrew-build-env
drwxr-xr-x 2 dimpase staff    4096 Apr  9 20:24 m4
-rw-r--r-- 1 dimpase staff    9744 Apr  9 20:24 Makefile
-rw-r--r-- 1 dimpase staff   20890 Apr  9 20:24 README.md
drwxr-xr-x 4 dimpase staff    4096 Apr  9 20:24 src
-rw-r--r-- 1 dimpase staff      51 Apr  9 20:24 VERSION.txt

comment:68 Changed 6 months ago by git

  • Commit changed from 1b03fa9a4f14439d0dbc3bd1af194ab25fa9eeb8 to 278c36c6caf9dfa8e7d33e0b45309615ee13a163

Branch pushed to git repo; I updated commit sha1. New commits:

b3b54b7Merge tag '9.3.rc2' into t/29039/pip_installable_package_sage_bootstrap
ec10bc9src/pkgs/sage_conf-pypi/setup.py: Fix for shell behavior on fedora
278c36csrc/pkgs/sage_conf-pypi/setup.py: Use 'configure --enable-build-as-root' so it works in containers too

comment:69 Changed 6 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:70 follow-up: Changed 6 months ago by dimpase

the dependent ticket is not merged in the branch, any reason for this?

comment:71 Changed 6 months ago by git

  • Commit changed from 278c36c6caf9dfa8e7d33e0b45309615ee13a163 to 94b8d3c7223785fab507ecbe38f9c16fb44a46de

Branch pushed to git repo; I updated commit sha1. New commits:

04da2c6build/pkgs/ipywidgets: Patch out declaring install-requires of nbformat and jupyterlab-widgets
94b8d3cMerge #30913

comment:72 in reply to: ↑ 70 Changed 6 months ago by mkoeppe

Replying to dimpase:

the dependent ticket is not merged in the branch, any reason for this?

Thanks for catching this, the latest version was not merged

comment:73 Changed 6 months ago by mkoeppe

Here on the ticket I am creating things in src/pkgs/ but in #31577 I have a different proposal for the source layout, comments welcome

comment:74 Changed 6 months ago by dimpase

now I get a failure in building numpy, if I run ./bootstrap && (cd src/pkgs/sage_conf-pypi/ && tox -v -v -v) after distclean.

Uninstalling 'numpy' with legacy uninstaller
Package 'numpy' is currently not installed
Traceback (most recent call last):
  File "../lapack_conf.py", line 3, in <module>
    import pkgconfig, os
ModuleNotFoundError: No module named 'pkgconfig'

and also weird error with cysignal:

checking for std::atomic with OpenMP in C++ code... yes
configure: creating ./config.status
config.status: creating src/cysignals/signals.pxd
config.status: creating src/config.h
config.status: creating src/cysignals/cysignals_config.h
configure: WARNING: unrecognized options: --disable-maintainer-mode, --disable-dependency-tracking
Installing cysignals-1.10.3
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fda2cad2310>, 'Connection to 192.0.2.0 timed out. (connect timeout=15)')': /simple/cython/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fda2cad2690>, 'Connection to 192.0.2.0 timed out. (connect timeout=15)')': /simple/cython/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fda2cad2890>, 'Connection to 192.0.2.0 timed out. (connect timeout=15)')': /simple/cython/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fda2cad2ad0>, 'Connection to 192.0.2.0 timed out. (connect timeout=15)')': /simple/cython/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fda2cad2d10>, 'Connection to 192.0.2.0 timed out. (connect timeout=15)')': /simple/cython/
ERROR: Could not find a version that satisfies the requirement Cython>=0.28
ERROR: No matching distribution found for Cython>=0.28
Traceback (most recent call last):
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
    subprocess.check_call(cmd)
  File "/usr/lib64/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu/local/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/home/scratch2/dimpase/tmp/tmp772nb9rh', '--quiet', 'Cython>=0.28']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "setup.py", line 152, in <module>
    cmdclass=dict(build=build, bdist_egg=no_egg),
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/setuptools/__init__.py", line 152, in setup
    _install_setup_requires(attrs)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/setuptools/dist.py", line 724, in fetch_build_eggs
    replace_conflicting=True,
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pkg_resources/__init__.py", line 768, in resolve
    replace_conflicting=replace_conflicting
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1051, in best_match
    return self.obtain(req, installer)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1063, in obtain
    return installer(requirement)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/setuptools/dist.py", line 780, in fetch_build_egg
    return fetch_build_egg(self, req)
  File "/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.7/site-packages/setuptools/installer.py", line 77, in fetch_build_egg
    raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu/local/bin/python3', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/home/scratch2/dimpase/tmp/tmp772nb9rh', '--quiet', 'Cython>=0.28']' returned non-zero exit status 1.
Error: could not determine package name

dependencies issue?

comment:75 Changed 6 months ago by mkoeppe

Note that make distclean does not clean stuff in .tox, so if you want to start from scratch, use tox -r

comment:76 Changed 6 months ago by dimpase

still, after successful make build, running the command in the ticket description ends with

  Error building Sage.

  The following package(s) may have failed to build (not necessarily
  during this run of 'make all-build'):

  * package:         numpy-1.19.5
    last build time: Apr 11 22:40
    log file:        /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu/logs/pkgs/numpy-1.19.5.log
    build directory: /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu/local/var/tmp/sage/build/numpy-1.19.5

and in the numpy log I see

Uninstalling 'numpy' with legacy uninstaller
Package 'numpy' is currently not installed
Traceback (most recent call last):
  File "../lapack_conf.py", line 3, in <module>
    import pkgconfig, os
ModuleNotFoundError: No module named 'pkgconfig'

real    0m0.064s
user    0m0.049s
sys     0m0.010s
************************************************************************
Error installing package numpy-1.19.5

a race condition, pkgconfig is not installed yet?

comment:77 Changed 6 months ago by dimpase

and this

$ ./sage --python
Python 3.7.7 (default, Mar 13 2020, 21:39:43) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkgconfig
>>> 

is OK. Is something weird happening with the Python while building numpy?

comment:78 Changed 6 months ago by dimpase

Oh, I see. The script lapack_conf.py calls sage-bootstrap-python, which is not Sage's python in this case, it's a python2.

clpc171[/home/scratch2/dimpase/sage/sage]$ ./sage --buildsh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/scratch2/dimpase/sage/sage
(sage-buildsh) dimpase@clpc171:sage$ sage-bootstrap-python 
Python 2.7.18 (default, Apr 21 2020, 18:49:31) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

comment:79 Changed 6 months ago by dimpase

  • Status changed from needs_review to needs_work

comment:80 Changed 6 months ago by mkoeppe

That's really something that we need to fix in a separate ticket!

comment:81 follow-up: Changed 6 months ago by mkoeppe

No, actually spkg-install.in calls python3.

comment:82 follow-up: Changed 6 months ago by mkoeppe

You are interacting with the wrong SAGE_ROOT, by the way. The relevant SAGE_ROOT is /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu. What's happening in /home/scratch2/dimpase/sage/sage is irrelevant.

comment:83 in reply to: ↑ 81 Changed 6 months ago by dimpase

Replying to mkoeppe:

No, actually spkg-install.in calls python3.

well, it calls the failing lapack_conf.py, which prescribes a different Python.

comment:84 Changed 6 months ago by mkoeppe

See #31663, where I explain this in more detail

comment:85 in reply to: ↑ 82 Changed 6 months ago by dimpase

Replying to mkoeppe:

You are interacting with the wrong SAGE_ROOT, by the way. The relevant SAGE_ROOT is /home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu. What's happening in /home/scratch2/dimpase/sage/sage is irrelevant.

OK, so I guess here is the test showing that pkgconfig module is not there (I cd to the said SAGE_ROOT and run ./local/bin/python3 there):

clpc171[/home/scratch2/dimpase/sage/sage/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3.rc2-cpython-37m-x86_64-linux-gnu]$ ./local/bin/python3
Python 3.7.7 (default, Mar 13 2020, 21:39:43) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkgconfig
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pkgconfig'

(and the same import works for the "usual" SAGE_ROOT just fine)

comment:86 Changed 6 months ago by dimpase

If I pip-install pkgconfig and cython in the correct SAGE_ROOT, then tox run gets through numpy install just fine, but fails later in a similar way on pplpy.

I guess dependency resolution is shot, or there is a race somewhere, leading to wrong order of installations.

comment:87 Changed 6 months ago by mkoeppe

Thanks for testing! This is very strange...

comment:88 Changed 6 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

comment:89 Changed 5 months ago by git

  • Commit changed from 94b8d3c7223785fab507ecbe38f9c16fb44a46de to df2f8d11a2d204b5e5197b968ab1bb8e62486508

Branch pushed to git repo; I updated commit sha1. New commits:

df2f8d1Merge tag '9.3' into t/29039/pip_installable_package_sage_bootstrap

comment:90 Changed 5 months ago by mkoeppe

  • Status changed from needs_work to needs_review

It may be worth testing whether at least the built source distribution (https://pypi.org/project/sagemath-standard/) works for you as advertised.

comment:91 follow-up: Changed 5 months ago by dimpase

it seems to parallelize make too aggressively. Just had to reboot a machine I tried this on. Namely, top shows a massive number of make instances launched, hundreds. The log is full of

make[20]: warning: -j4 forced in submake: resetting jobserver mode.
  make -j4 build/make/Makefile --stop

Also, the 1st attempt to run tox -v -v -v failed with a message like "packages PEP <something> are not directly installable" (or something like this). I just started it again, and it went on.

comment:92 Changed 5 months ago by mkoeppe

  • Description modified (diff)

comment:93 in reply to: ↑ 91 Changed 5 months ago by mkoeppe

Replying to dimpase:

it seems to parallelize make too aggressively. Just had to reboot a machine I tried this on.

This sounds more like an infinite "make" recursion. Could you remove the setting of -j from src/pkgs/sage_conf-pypi/setup.py (and from your environment) and try what it does then?

comment:94 Changed 5 months ago by dimpase

I still quickly get make completely taking over the machine:

$ ps -def | grep make | wc -l
1797

even after

  • src/pkgs/sage_conf-pypi/setup.py

    diff --git a/src/pkgs/sage_conf-pypi/setup.py b/src/pkgs/sage_conf-pypi/setup.py
    index b7646224d6..8ee7211104 100644
    a b class build_py(distutils_build_py): 
    5252        # but then a user could use "make build-venv" to build compatible wheels for all Python packages.
    5353        # TODO: A target to only build wheels of tricky packages
    5454        # (that use native libraries shared with other packages).
    55         SETMAKE = 'if [ -z "$MAKE" ]; then export MAKE="make -j$(PATH=build/bin:$PATH build/bin/sage-build-num-threads | cut -d" " -f 2)"; fi'
     55        SETMAKE = 'if [ -z "$MAKE" ]; then export MAKE="make"; fi'
    5656        cmd = f'cd {SAGE_ROOT} && {SETENV} && {SETMAKE} && $MAKE V=0 build'
    5757        if os.system(cmd) != 0:
    5858            raise DistutilsSetupError("make build-local failed")

no more warning: -j4 forced in submake: resetting jobserver mode seen, but that's not the root cause. This is with make 4.3.

comment:95 Changed 5 months ago by dimpase

It could be that some wrong links are created by git; if I try to pull this branch I get

CONFLICT (directory/file): There is a directory with name build/pkgs/sage_conf/src in HEAD. Adding build/pkgs/sage_conf/src as build/pkgs/sage_conf/src~df2f8d11a2d204b5e5197b968ab1bb8e62486508
Automatic merge failed; fix conflicts and then commit the result.

and perhaps I didn't fix conflicts correctly.

It would be good to have something rebased on the develop branch here.

comment:96 Changed 5 months ago by dimpase

OK, on a clean tree checkout I don't get the make issues. Apologies.

comment:97 Changed 5 months ago by dimpase

./bootstrap && (cd src/pkgs/sage_conf-pypi/ && tox -r -v -v -v) seems to work, although I had to restart tox -r -v -v -v at some moment. Retrying after distclean, to see if I can reproduce this. I'm getting

 make --no-print-directory polytopes_db-no-deps
  sage-logger -p 'SAGE_CHECK=no sage-spkg   polytopes_db-20170220.p0' '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/polytopes_db-20170220.p0.log'
  [polytopes_db-20170220.p0] installing. Log file: /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/polytopes_db-20170220.p0.log
    [polytopes_db-20170220.p0] successfully installed.
  make --no-print-directory singular-no-deps
  sage-logger -p 'SAGE_CHECK=no sage-spkg   singular-4.2.0p1+2021-04-06+sage' '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/singular-4.2.0p1+2021-04-06+sage.log'
  [singular-4.2.0p1+2021-04-06+sage] installing. Log file: /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/singular-4.2.0p1+2021-04-06+sage.log
    [palp-2.11] successfully installed.
  make --no-print-directory ratpoints-no-deps
  make[3]: *** No rule to make target '../pkgs/sage_conf/src/bin/sage-env-config', needed by '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/local/var/lib/sage/installed/sage_conf-9.3'.  Stop.
  make[3]: *** Waiting for unfinished jobs....
  sage-logger -p 'SAGE_CHECK=no sage-spkg   ratpoints-2.1.3.p5' '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/ratpoints-2.1.3.p5.log'
  [ratpoints-2.1.3.p5] installing. Log file: /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/ratpoints-2.1.3.p5.log
    [ratpoints-2.1.3.p5] successfully installed.
    [ecl-21.2.1] successfully installed.
    [gap-4.11.0.p1] successfully installed.
    [singular-4.2.0p1+2021-04-06+sage] successfully installed.
  make[2]: *** [Makefile:2150: all-build] Error 2
  make[2]: Leaving directory '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/build/make'

  real	9m41.134s
  user	24m26.005s
  sys	1m57.351s
  ***************************************************************
  Error building Sage.

  The following package(s) may have failed to build (not necessarily
  during this run of 'make all-build'):

  It is safe to delete any log files and build directories, but they
  contain information that is helpful for debugging build problems.
  WARNING: If you now run 'make' again, the build directory of the
  same version of the package will, by default, be deleted. Set the
  environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

  make[1]: *** [Makefile:39: all-build] Error 1
  make[1]: Leaving directory '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu'
  make: *** [Makefile:16: build] Error 2
  Running cd /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu && (. ./.homebrew-build-env 2> /dev/null || :) && ./configure --prefix=/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/local --with-python=/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python --enable-build-as-root --with-system-python3=force --disable-notebook --disable-sagelib
  error: make build-local failed
  Building wheel for sage-conf (PEP 517) ... error
  ERROR: Failed building wheel for sage-conf
Failed to build sage-conf
ERROR: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 358, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.
You should consider upgrading via the '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --upgrade pip' command.
Removed build tracker: '/tmp/pip-req-tracker-edpp7niq'
ERROR: invocation failed (exit code 1)
python finish: installpkg /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3.zip after 622.22 seconds
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
ERROR:   python: InvocationError for command /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --exists-action w -v .tox/.tmp/package/1/sage_conf-9.3.zip (exited with code 1)
cleanup /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3.zip

At this point I can just start tox -r -v -v -v again, and it happily continues. Not sure whether the final result is correct.

Anyway, the error isn't good. Looks like a bit of an error in a make rule, it tries to make ../pkgs/sage_conf/src/bin/sage-env-config.

comment:98 Changed 5 months ago by dimpase

After the restart, I see again

make[3]: *** No rule to make target '../pkgs/sage_conf/src/bin/sage-env-config', needed by '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/local/var/lib/sage/installed/sage_conf-9.3'.  Stop.
  make[3]: *** Waiting for unfinished jobs....

but it does not lead to (immediate) termination.

comment:99 Changed 5 months ago by dimpase

after a while it then errors out with already mentioned

Failed to build sage-conf
ERROR: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly

in more detail:

...
  [singular-4.2.0p1+2021-04-06+sage] successfully installed.
  make[2]: *** [Makefile:2150: all-build] Error 2
  make[2]: Leaving directory '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/build/make'

  real	10m35.056s
  user	24m37.840s
  sys	2m0.611s
  ***************************************************************
  Error building Sage.

  The following package(s) may have failed to build (not necessarily
  during this run of 'make all-build'):

  It is safe to delete any log files and build directories, but they
  contain information that is helpful for debugging build problems.
  WARNING: If you now run 'make' again, the build directory of the
  same version of the package will, by default, be deleted. Set the
  environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

  make[1]: *** [Makefile:39: all-build] Error 1
  make[1]: Leaving directory '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu'
  make: *** [Makefile:16: build] Error 2
  Running cd /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu && (. ./.homebrew-build-env 2> /dev/null || :) && ./configure --prefix=/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/local --with-python=/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python --enable-build-as-root --with-system-python3=force --disable-notebook --disable-sagelib
  error: make build-local failed
  Building wheel for sage-conf (PEP 517) ... error
  ERROR: Failed building wheel for sage-conf
Failed to build sage-conf
ERROR: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 358, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.
You should consider upgrading via the '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --upgrade pip' command.
Removed build tracker: '/tmp/pip-req-tracker-u1_zy58f'
ERROR: invocation failed (exit code 1)
python finish: installpkg /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3.zip after 675.89 seconds
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
ERROR:   python: InvocationError for command /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --exists-action w -v .tox/.tmp/package/1/sage_conf-9.3.zip (exited with code 1)
cleanup /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3.zip

comment:100 Changed 5 months ago by dimpase

  • Status changed from needs_review to needs_work

No, no matter how many times I restart tox, it errors out with

make --no-print-directory singular-no-deps
  sage-logger -p 'SAGE_CHECK=no sage-spkg   singular-4.2.0p1+2021-04-06+sage' '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/singular-4.2.0p1+2021-04-06+sage.log'
  [singular-4.2.0p1+2021-04-06+sage] installing. Log file: /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/singular-4.2.0p1+2021-04-06+sage.log
    [palp-2.11] successfully installed.
  make --no-print-directory ratpoints-no-deps
  make[3]: *** No rule to make target '../pkgs/sage_conf/src/bin/sage-env-config', needed by '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/local/var/lib/sage/installed/sage_conf-9.3'.  Stop.
  make[3]: *** Waiting for unfinished jobs....
  sage-logger -p 'SAGE_CHECK=no sage-spkg   ratpoints-2.1.3.p5' '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/ratpoints-2.1.3.p5.log'
  [ratpoints-2.1.3.p5] installing. Log file: /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/logs/pkgs/ratpoints-2.1.3.p5.log
    [ratpoints-2.1.3.p5] successfully installed.
    [ecl-21.2.1] successfully installed.
    [gap-4.11.0.p1] successfully installed.
    [singular-4.2.0p1+2021-04-06+sage] successfully installed.
  make[2]: *** [Makefile:2150: all-build] Error 2
  make[2]: Leaving directory '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/build/make'

  real	9m16.619s
  user	23m45.919s
  sys	1m54.965s
  ***************************************************************
  Error building Sage.

  The following package(s) may have failed to build (not necessarily
  during this run of 'make all-build'):

  It is safe to delete any log files and build directories, but they
  contain information that is helpful for debugging build problems.
  WARNING: If you now run 'make' again, the build directory of the
  same version of the package will, by default, be deleted. Set the
  environment variable SAGE_KEEP_BUILT_SPKGS=yes to prevent this.

  make[1]: *** [Makefile:39: all-build] Error 1
  make[1]: Leaving directory '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu'
  make: *** [Makefile:16: build] Error 2
  Running cd /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu && (. ./.homebrew-build-env 2> /dev/null || :) && ./configure --prefix=/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/.sage/sage-9.3-cpython-38-x86_64-linux-gnu/local --with-python=/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python --enable-build-as-root --with-system-python3=force --disable-notebook --disable-sagelib
  error: make build-local failed
  Building wheel for sage-conf (PEP 517) ... error
  ERROR: Failed building wheel for sage-conf
Failed to build sage-conf
ERROR: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
Exception information:
Traceback (most recent call last):
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 189, in _main
    status = self.run(options, args)
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 358, in run
    raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for sage-conf which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 21.0.1; however, version 21.1.1 is available.
You should consider upgrading via the '/mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --upgrade pip' command.
Removed build tracker: '/tmp/pip-req-tracker-9ti8dn54'
ERROR: invocation failed (exit code 1)
python finish: installpkg /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3.zip after 597.56 seconds
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ summary __________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
ERROR:   python: InvocationError for command /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/python/bin/python -m pip install --exists-action w -v .tox/.tmp/package/1/sage_conf-9.3.zip (exited with code 1)
cleanup /mnt/opt/Sage/sage-clang/src/pkgs/sage_conf-pypi/.tox/.tmp/package/1/sage_conf-9.3.zip

comment:101 Changed 5 months ago by git

  • Commit changed from df2f8d11a2d204b5e5197b968ab1bb8e62486508 to b4b3ec80489aa21324724c2c7ead38d37061fa00

Branch pushed to git repo; I updated commit sha1. New commits:

b4b3ec8Merge tag '9.4.beta0' into t/29039/pip_installable_package_sage_bootstrap

comment:102 Changed 5 months ago by git

  • Commit changed from b4b3ec80489aa21324724c2c7ead38d37061fa00 to e1d2df08aa14e47a780c51c01073d58f4c7b3426

Branch pushed to git repo; I updated commit sha1. New commits:

e1d2df0Merge tag '9.4.beta1' into t/29039/pip_installable_package_sage_bootstrap

comment:103 Changed 5 months ago by mkoeppe

  • Dependencies #30913 deleted

comment:104 Changed 4 months ago by git

  • Commit changed from e1d2df08aa14e47a780c51c01073d58f4c7b3426 to 900e621eca31b625831b14918d1a34cf61884d07

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

b96fd8bAdd configure option --disable-sagelib
900e621Add sage_root to sage_bootstrap

comment:105 Changed 4 months ago by mkoeppe

  • Dependencies set to #31577

comment:106 Changed 4 months ago by git

  • Commit changed from 900e621eca31b625831b14918d1a34cf61884d07 to 89bdadf349474996b7a4c786b82471273eeb1c4d

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

40abff0pkgs/sage-setup/setup.cfg: README.rst, not README.md
782b64bpkgs/sagemath-standard: Remove symlink to SAGE_ROOT/src/sage_setup
cf606f7build/bin/write-dockerfile.sh: ADD src/sage_setup before building TARGETS_PRE
a1b8832build/pkgs/sage_setup/spkg-src: Use python3
9270417tox.ini, build/bin/write-dockerfile.sh, tox*.yml: Use all-sage-local as default TARGETS_PRE
870f9aabuild/pkgs/memory_allocator/install-requires.txt: Fix filename
61c25a0build/pkgs/sage_conf/install-requires.txt: New
8d432caMerge #29847
26e16c7Add configure option --disable-sagelib
89bdadfAdd sage_root to sage_bootstrap

comment:107 Changed 4 months ago by git

  • Commit changed from 89bdadf349474996b7a4c786b82471273eeb1c4d to 3dc9618377133f7983a374630cd3459568a3a10a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3dc9618pkgs/sage-conf_pypi: New

comment:108 Changed 4 months ago by mkoeppe

  • Description modified (diff)

comment:109 Changed 4 months ago by git

  • Commit changed from 3dc9618377133f7983a374630cd3459568a3a10a to e825a9f554fcc4eda8665fa9b9f708d6382b324d

Branch pushed to git repo; I updated commit sha1. New commits:

ba67c92Put pkgs/sage-conf in the root
e825a9fpkgs/sage-conf_pypi: Fixup MANIFEST.in, setup.py

comment:110 Changed 4 months ago by mkoeppe

  • Dependencies changed from #31577 to #31577, #32057

comment:111 Changed 4 months ago by git

  • Commit changed from e825a9f554fcc4eda8665fa9b9f708d6382b324d to fe8019a075a5870c0ecbb032c34fc8579bdd90d9

Branch pushed to git repo; I updated commit sha1. New commits:

ad8bad1Makefile (pypi-sdists): New
0867811src/sage_setup/setenv.py: Only set ...PATH variables and LDFLAGS if SAGE_LOCAL differs from SAGE_VENV
fe8019aMerge #32057

comment:112 Changed 4 months ago by git

  • Commit changed from fe8019a075a5870c0ecbb032c34fc8579bdd90d9 to 8c0ccf319cf038e836c38f494d380f30eb5413ca

Branch pushed to git repo; I updated commit sha1. New commits:

8c0ccf3src/pkgs/sage_conf*/README.rst: Unify the files

comment:113 Changed 4 months ago by git

  • Commit changed from 8c0ccf319cf038e836c38f494d380f30eb5413ca to 2181f10720e2c6531b33d84be0cc0aeb8e660030

Branch pushed to git repo; I updated commit sha1. New commits:

2181f10pkgs/sage-conf_pypi: Fixup

comment:114 Changed 4 months ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:115 Changed 4 months ago by git

  • Commit changed from 2181f10720e2c6531b33d84be0cc0aeb8e660030 to 44b1e8d88affb78b8c06f478f79d2414b4efa232

Branch pushed to git repo; I updated commit sha1. New commits:

44b1e8dMakefile (pypi-sdists): Add dependency on sage_setup

comment:116 Changed 4 months ago by mkoeppe

  • Dependencies changed from #31577, #32057 to #31577, #32057, #29847

comment:117 Changed 4 months ago by mkoeppe

  • Dependencies changed from #31577, #32057, #29847 to #31577, #32057, #29847, #32071

comment:118 Changed 4 months ago by dimpase

I tried testing this over a recent pynac 0.7.28 branch, and it errors out as it apparently does not know how to pull upstream tarballs not from the mirrors. Missing --enable-download-from-upstream-url missing somewhere?

comment:119 Changed 4 months ago by git

  • Commit changed from 44b1e8d88affb78b8c06f478f79d2414b4efa232 to 1596cb58ab2eb42e62edabc5e94c6bf9e15a46ac

Branch pushed to git repo; I updated commit sha1. New commits:

1596cb5pkgs/sage-conf_pypi/setup.py: Use --enable-download-from-upstream-url

comment:120 Changed 4 months ago by dimpase

  • Status changed from needs_review to positive_review

OK, this works

comment:121 Changed 4 months ago by mkoeppe

Thank you!

comment:122 Changed 4 months ago by git

  • Commit changed from 1596cb58ab2eb42e62edabc5e94c6bf9e15a46ac to 1dd45a23d5fca5fecb4bc9fdb840e3b337e16323
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

380b42dUpdate paths in comments
a7d6a3ebuild/bin/write-dockerfile.sh: ADD pkgs
7568dc6Merge tag '9.4.beta4' into t/31577/move_sage_root_build_pkgs___src_to_sage_root____rename_sage_root_build_to_sage_root_sage_bootstrap
1dd45a2Merge #31577

comment:123 Changed 4 months ago by git

  • Commit changed from 1dd45a23d5fca5fecb4bc9fdb840e3b337e16323 to 569b188b46fc42a015af7c30fdf0b3699ccd4c19

Branch pushed to git repo; I updated commit sha1. New commits:

7f0bd4aMerge tag '9.4.beta4' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
569b188Merge #29847

comment:124 Changed 4 months ago by mkoeppe

  • Status changed from needs_review to positive_review

comment:125 Changed 3 months ago by git

  • Commit changed from 569b188b46fc42a015af7c30fdf0b3699ccd4c19 to cfd0c254d78f2f5fb5004e168235e5b520e43717
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

35b8b74Merge tag '9.4.beta5' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
773ef29Merge #32073
9aa1f10Merge #29585
8c5ee65build/pkgs/pip/install-requires.txt: Update lower bound
ed5f9f0Merge tag '9.4.beta6' into t/32046/use_pip___use_feature_in_tree_build__replace_use_of_sdh_setup_bdist_wheel_by_sdh_pip_install
6b1b222Merge #32046
cfd0c25Merge #29847

comment:126 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

comment:127 Changed 2 months ago by mkoeppe

  • Dependencies changed from #31577, #32057, #29847, #32071 to #29847, #32371

comment:128 Changed 2 months ago by git

  • Commit changed from cfd0c254d78f2f5fb5004e168235e5b520e43717 to 7f13d04a0a80e3321348d7a172f11c91dce47fb4

Branch pushed to git repo; I updated commit sha1. New commits:

85d1d27Merge tag '9.4.rc1' into t/32046/use_pip___use_feature_in_tree_build__replace_use_of_sdh_setup_bdist_wheel_by_sdh_pip_install
050f8dfbuild/pkgs/numpy/spkg-install.in: Use sdh_pip_install, sdh_prefix_args
a41b507Merge branch 't/32046/use_pip___use_feature_in_tree_build__replace_use_of_sdh_setup_bdist_wheel_by_sdh_pip_install' into t/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup
e6e8cedMerge #29847
d69c496build/pkgs/ptyprocess: pin version to 0.5.1
7f13d04Merge #32371

comment:129 Changed 2 months ago by git

  • Commit changed from 7f13d04a0a80e3321348d7a172f11c91dce47fb4 to b011389172f7ebce9066a33dd054f038e322130a

Branch pushed to git repo; I updated commit sha1. New commits:

290fa2fbuild/pkgs/sagelib/src: Restore lost symlink
478ffcdMerge #32046
b011389Merge branch 't/29847/install_sage_setup_with_pip__move_sage_include_directories__cython_aliases_from_sage_env_to_sage_setup' into t/29039/pip_installable_package_sage_bootstrap

comment:130 Changed 2 months ago by git

  • Commit changed from b011389172f7ebce9066a33dd054f038e322130a to e38644a57e2aec25c876a2ef82744e38fa14b396

Branch pushed to git repo; I updated commit sha1. New commits:

e38644aMakefile (pypi-sdists): Add sage_setup

comment:131 Changed 2 months ago by git

  • Commit changed from e38644a57e2aec25c876a2ef82744e38fa14b396 to bad918919b3b4ac2a95a9c9e2d4283fd43589791

Branch pushed to git repo; I updated commit sha1. New commits:

b2368c6pkgs/sage-setup/setup.cfg: Add pkgconfig as install_requires
bad9189Merge #29847

comment:132 Changed 5 weeks ago by mkoeppe

To make it more user-friendly, on a failing configure it should display install instructions for build prereqs.

On a fresh Google Colab:

!pip install -v sage-conf
...
  configure: Sorry, the 'm4' command must be in the path to build Sage
  configure: On some systems it can be found in /usr/ccs/bin
  configure: See also http://www.gnu.org/software/m4/
  configure: error: Exiting, as the macro processor 'm4' can not be found.
  Running cd /root/.sage/sage-9.4-cpython-37m-x86_64-linux-gnu && (. ./.homebrew-build-env 2> /dev/null || :) && ./configure --prefix=/root/.sage/sage-9.4-cpython-37m-x86_64-linux-gnu/local --with-python=/usr/bin/python3 --enable-build-as-root --enable-download-from-upstream-url --with-system-python3=force --disable-notebook --disable-sagelib
  error: configure failed
  Building wheel for sage-conf (PEP 517) ... error
  ERROR: Failed building wheel for sage-conf
Failed to build sage-conf

comment:133 Changed 5 weeks ago by git

  • Commit changed from bad918919b3b4ac2a95a9c9e2d4283fd43589791 to a2fd62006ef6e6b02288f7122bd390f980f97bbb

Branch pushed to git repo; I updated commit sha1. New commits:

0d3df50Merge tag '9.4' into t/29039/pip_installable_package_sage_bootstrap
a2fd620pkgs/sage-conf_pypi/setup.py: Show _prereq if configure fails

comment:134 Changed 5 weeks ago by git

  • Commit changed from a2fd62006ef6e6b02288f7122bd390f980f97bbb to f9d6a72fa91c2e1aaeb9962c14ccbfcb49d2f92c

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f9d6a72pkgs/sage-conf_pypi/setup.py: Show _prereq if configure fails

comment:135 Changed 5 weeks ago by mkoeppe

Better now... but would be better with deferred configure exits (#32060).

  -----------------------------------------------------------------------------
  Checking whether SageMath should install SPKG python3...
  checking whether any of bzip2 xz libffi is installed as or will be installed as SPKG... yes; install python3 as well
  configure: error: given --with-system-python3=force but no system package could be used
  configure failed; this may be caused by missing build prerequisites.
  You can install the required build prerequisites using the following shell command

  sudo apt-get install  binutils make m4 perl python3 tar bc gcc g++ ca-certificates

comment:136 Changed 5 weeks ago by git

  • Commit changed from f9d6a72fa91c2e1aaeb9962c14ccbfcb49d2f92c to cfdcad4fead8584aeabd193880011dbfb87374d2

Branch pushed to git repo; I updated commit sha1. New commits:

cfdcad4pkgs/sage-conf_pypi/setup.py: Include more packages in hint if configure fails

comment:137 Changed 5 weeks ago by mkoeppe

  • Dependencies #29847, #32371 deleted

comment:138 Changed 5 weeks ago by git

  • Commit changed from cfdcad4fead8584aeabd193880011dbfb87374d2 to 04abd6545d47573faa3d4d571b2176b13e17558d

Branch pushed to git repo; I updated commit sha1. New commits:

04abd65pkgs/sage-conf/README.rst: Update directories

comment:139 Changed 4 weeks ago by mkoeppe

  • Dependencies set to #32060

comment:140 Changed 4 weeks ago by git

  • Commit changed from 04abd6545d47573faa3d4d571b2176b13e17558d to 2622b71332368c1726d7df4641bdb255de277058

Branch pushed to git repo; I updated commit sha1. New commits:

b566863m4/sage_spkg_{collect,configure}.m4: Defer error message from --with-system-SPKG=force to the end
48cdeb4configure.ac, sage_spkg_configure.m4: Set default --with-system-gcc=force, --with-system-python3=force
8e92b98tox.ini: Override new configure --with-system-python3=force default from #32060
19efe21configure.ac: Revert the change to the default for --with-system-python3
2622b71Merge #32060

comment:141 Changed 4 weeks ago by mkoeppe

  • Reviewers changed from Dima Pasechnik to Dima Pasechnik, ...
Last edited 4 weeks ago by mkoeppe (previous) (diff)

comment:142 Changed 4 weeks ago by git

  • Commit changed from 2622b71332368c1726d7df4641bdb255de277058 to 0cdba5560af1cd6361a881249c794e08a65c6c37

Branch pushed to git repo; I updated commit sha1. New commits:

0cdba55Merge tag '9.5.beta2' into t/29039/pip_installable_package_sage_bootstrap

comment:143 Changed 4 weeks ago by mkoeppe

  • Dependencies #32060 deleted

comment:144 Changed 4 weeks ago by mkoeppe

  • Dependencies set to #32442
  • Status changed from needs_review to needs_work

comment:145 Changed 4 weeks ago by git

  • Commit changed from 0cdba5560af1cd6361a881249c794e08a65c6c37 to 9dcc18938eea954544864ba20f87d88c5366a50c

Branch pushed to git repo; I updated commit sha1. New commits:

a23b1e1pkgs/sage-conf_pypi/MANIFEST.in: Exclude build artifacts
2f5be2dconfigure.ac, build/pkgs/python3/spkg-configure.m4: New default --with-sage-venv=auto
f700640configure.ac: In config.status, create convenience symlinks prefix, venv
b4a3845m4/sage_spkg_collect.m4: If we install python3 spkg, install it in SAGE_VENV
12cbbdaconfigure.ac: Keep old behavior of SAGE_VENV in incremental builds
117fe3abuild/make/Makefile.in (...-clean): Do not fail trying to uninstall if SAGE_VENV has not been created yet
ce1c344Merge #32442
e71b8a6pkgs/sage-conf_pypi/setup.py: Key SAGE_ROOT only to Sage version and architecture
9dcc189pkgs/sage-conf_pypi/setup.py: Use --with-sage-venv

comment:146 Changed 4 weeks ago by mkoeppe

  • Description modified (diff)

comment:147 Changed 4 weeks ago by mkoeppe

  • Work issues set to fix up sage_conf build location

comment:148 Changed 3 weeks ago by fbissey

A little bit late at this stage to ask for stuff but here it goes. Would it be appropriate to split the file sage_conf.py into two parts?

One that would keep the name and contains the main function and doesn't need to be configured, and the second one that just contains the variables and that would be pre-parsed by configure.

The idea being that distro could just provide/configure the second file has they see fit without touching the functionality of the package. Although I am not sure what to do with sage-env-config yet in that scenario.

comment:149 Changed 3 weeks ago by mkoeppe

Yes, this makes sense but I think it would be best done in a follow-up ticket to this ticket and to #31396. Basically a bit of refactoring for the three versions of sage_conf.

comment:150 Changed 3 weeks ago by fbissey

I can wait for that but that would go a long way to ease my adopting it as a new package.

I am working on splitting sage in sage-on-gentoo on similar lines with some success. I now have a separate sage_setup and sage_docbuild -mostly with the idea that they can be usable by downstream sage package like flatsurf and snappy that I am looking at packaging experimentally. And in the next beta we can expect to have a separate sage-doc to debut. Well "re-debut", it used to be a separate package 10 years ago before I brought back everything in a single coherent unit. Fells a bit like a trip back in time but we are definitely not in the same place as then.

comment:151 Changed 3 weeks ago by git

  • Commit changed from 9dcc18938eea954544864ba20f87d88c5366a50c to c695921c3cee61a32a8d7ec9443bb4e315eb481b

Branch pushed to git repo; I updated commit sha1. New commits:

d626aa8build/pkgs/pplpy/dependencies: Depend on sphinx only if SAGE_SPKG_INSTALL_DOCS!=no
82ce539configure.ac: If --disable-notebook and --disable-sagelib, do not build zeromq etc.
02505capkgs/sage-conf_pypi/MANIFEST.in: Update from #31396
aeca1a0Makefile (pypi-sdists): Add sage_setup
cd3dce9pkgs/sage-conf_pypi/setup.py: Update directory of configured sage_conf.py
c695921pkgs/sage-conf/sage_conf.py.in (SAGE_SPKG_WHEELS): Update so it works if SAGE_VENV != SAGE_LOCAL

comment:152 Changed 3 weeks ago by mkoeppe

  • Status changed from needs_work to needs_review
  • Work issues fix up sage_conf build location deleted

comment:153 Changed 10 days ago by mkoeppe

  • Description modified (diff)
Note: See TracTickets for help on using tickets.