Opened 3 years ago

Last modified 5 months ago

#29041 needs_work enhancement

sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.7
Component: build Keywords: sd111
Cc: slabbe, dimpase, embray, vbraun, fbissey, thansen, gh-tobiasdiez Merged in:
Authors: Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

We extend the sage_bootstrap package to add the commands:

  • sage -package list --output=install-requires PACKAGES... (for #30913)
  • sage -package list --output=requirements PACKAGES...
  • sage -package list --output=pipfile PACKAGES...
  • sage -package list --output=debian PACKAGE... etc. (taking over the functionality of the shell script build/bin/sage-get-system-packages.)

They generate text output in the format of setup.cfg [options] install_requires=, requirements.txt, and Pipfile; or a list of system packages

bootstrap can use these commands to generate these files for build/pkgs/sagelib/src/ or src. They are deleted by make bootstrap-clean.


References:

Change History (46)

comment:1 Changed 3 years ago by mkoeppe

  • Branch set to u/mkoeppe/at___bootstrap_time__generate_src_requirements_txt__src_constraints_txt__src_setup_cfg__install_requires__from_build_pkgs

comment:2 Changed 3 years ago by mkoeppe

  • Commit set to cfcc9fd4719a1bbba4bd763371fc6048c34d30ac
  • Status changed from new to needs_review

This is what it could look like - just for a small handful of packages for now


New commits:

cfcc9fdbootstrap: generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs

comment:3 follow-up: Changed 3 years ago by mkoeppe

The apt-get line in the install manual could be generated in the same way.

comment:4 Changed 3 years ago by mkoeppe

  • Authors set to Matthias Koeppe

comment:5 in reply to: ↑ 3 Changed 3 years ago by mkoeppe

Replying to mkoeppe:

The apt-get line in the install manual could be generated in the same way.

This is #29053.

comment:6 Changed 3 years ago by dimpase

I see that only few packages, e.g. pillow, are mentioned in the branch. Are they there as an example?

comment:7 Changed 3 years ago by mkoeppe

Yes, it's an initial example. More can be added on this ticket and/or on follow up tickets.

It may be the most efficient to first put this infrastructure (and the one for Debian) in place and then do invidual packages one by one to determine useful version bounds.

comment:8 Changed 3 years ago by dimpase

how is it going to play out with version constraints in spkg-configure.m4 files?

comment:9 Changed 3 years ago by mkoeppe

So far there are no spkg-configure.m4 files for any Python packages, and I would hope there will be no need for them.

comment:10 Changed 3 years ago by mkoeppe

Basically sagelib-install-requires.txt will play the same role for Python packages as spkg-configure.m4 does for non-Python packages, and likewise sagelib-requirements.txt : package-version.txt.

comment:11 Changed 3 years ago by mkoeppe

@embray If you have time - this ticket is waiting for review.

comment:12 Changed 3 years ago by embray

  • Status changed from needs_review to needs_work

Similar to what I wrote on #26964, I don't think this should be done in the bootstrap script. I don't want this script growing too far beyond its core purpose, which is simply to bootstrap the configure script.

This could be done by configure instead, using say a requirements.txt.in.

comment:13 Changed 3 years ago by mkoeppe

See my comments on #26964.

comment:14 Changed 3 years ago by mkoeppe

  • Status changed from needs_work to needs_review

comment:15 Changed 2 years ago by mkoeppe

  • Dependencies set to #29233

comment:16 Changed 2 years ago by git

  • Commit changed from cfcc9fd4719a1bbba4bd763371fc6048c34d30ac to 4fb30a608cd8b54be4fb0cc1a5b397b85480e5c6

Branch pushed to git repo; I updated commit sha1. New commits:

0dc97dbClean src/doc/en/installation/*.txt in bootstrap-clean, not misc-clean
bae1a82Merge branch 't/29233/bootstrap____make_distclean_broken' into t/29041/at___bootstrap_time__generate_src_requirements_txt__src_constraints_txt__src_setup_cfg__install_requires__from_build_pkgs
4fb30a6bootstrap: Add src/constraints.txt etc. to configure tarball

comment:17 Changed 2 years ago by git

  • Commit changed from 4fb30a608cd8b54be4fb0cc1a5b397b85480e5c6 to f147962f260f4ecf6872f3555111c9e8e300acea

Branch pushed to git repo; I updated commit sha1. New commits:

f147962src/bootstrap: Make executable

comment:18 Changed 2 years ago by mkoeppe

  • Cc vbraun added
  • Description modified (diff)

comment:19 Changed 2 years ago by mkoeppe

  • Cc fbissey added

comment:20 Changed 2 years ago by git

  • Commit changed from f147962f260f4ecf6872f3555111c9e8e300acea to 13088e302e305761994c04dca1d7b7d9d4b2f06d

Branch pushed to git repo; I updated commit sha1. New commits:

13088e3Merge tag '9.1.beta6' into t/29041/at___bootstrap_time__generate_src_requirements_txt__src_constraints_txt__src_setup_cfg__install_requires__from_build_pkgs

comment:21 Changed 2 years ago by mkoeppe

Needs review

comment:22 Changed 2 years ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:23 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-9.1 to sage-9.2

comment:24 Changed 2 years ago by mkoeppe

  • Cc thansen added

comment:25 Changed 2 years ago by mkoeppe

  • Dependencies changed from #29233 to #29233, #28925

comment:26 Changed 2 years ago by mkoeppe

  • Milestone changed from sage-9.2 to sage-9.3

comment:27 Changed 2 years ago by mkoeppe

  • Summary changed from at ./bootstrap time, generate src/requirements.txt, src/constraints.txt, src/setup.cfg [install_requires] from build/pkgs to at ./bootstrap time, generate build/pkgs/sagelib/src/requirements.txt, constraints.txt, setup.cfg [install_requires] from build/pkgs

comment:28 Changed 2 years ago by mkoeppe

  • Dependencies changed from #29233, #28925 to #29950
  • Work issues set to redo on top of #29950

comment:29 Changed 2 years ago by mkoeppe

  • Cc gh-tobiasdiez added
  • Description modified (diff)
  • Summary changed from at ./bootstrap time, generate build/pkgs/sagelib/src/requirements.txt, constraints.txt, setup.cfg [install_requires] from build/pkgs to at ./bootstrap time, generate requirements.txt, constraints.txt, setup.cfg [install_requires], Pipfile from build/pkgs

comment:30 follow-up: Changed 2 years ago by gh-tobiasdiez

Another approach would be to maintain an independent pipfile/requirements.txt in src/ and have automatic checks to make sure that the sage packages provide the required packages with a suitable version.

Reason: I feel like sage-lib should be independent of sage-distribution as much as possible.

More practical reason: pipenv works very nicely on Windows, and my hope is to use it without having to use sage make beforehand. So one simply clones the code, starts the virtual env with pipenv and gets an editiable install of sage for easy development. But for this the pipfile needs to be there without running the bootstrap command. (The alternative would be to rewrite the bootstrap cmd in a OS-independent way, say using powershell.)

comment:31 in reply to: ↑ 30 Changed 2 years ago by mkoeppe

Replying to gh-tobiasdiez:

Another approach would be to maintain an independent pipfile/requirements.txt in src/ and have automatic checks to make sure that the sage packages provide the required packages with a suitable version.

Reason: I feel like sage-lib should be independent of sage-distribution as much as possible.

I think the versions given by sage-the-distribution (build/pkgs/*/package-version.txt) should be considered as pinned versions, and information such as requirements.txt or Pipfile/Pipfile.lock should come from there in some way.

The allowed version ranges are currently not encoded anywhere. For non-Python packages have spkg-configure.m4 (#27330), but this is obviously the wrong mechanism for Python packages. setuptools has them in setup.cfg [install_requires].

I think I agree with you that this information does not necessarily have to be generated from (new) files in build/pkgs/ but could be put directly into src/, such as src/setup.cfg.

More practical reason: pipenv works very nicely on Windows, and my hope is to use it without having to use sage make beforehand. So one simply clones the code, starts the virtual env with pipenv and gets an editiable install of sage for easy development.

I think some steps are missing here - we don't have a native Windows build of many packages that Sage depends on. But I agree, after modularization (#29705) we should aim for getting a native Windows build of a Sage core at least.

But for this the pipfile needs to be there without running the bootstrap command. (The alternative would be to rewrite the bootstrap cmd in a OS-independent way, say using powershell.)

Note that we have the bootstrap -d mechanism, which downloads a tarball of the files created by bootstrap. This is how most users and developers get these files. We could look into creating a powershell version of bootstrap -d.

comment:32 Changed 2 years ago by mkoeppe

  • Description modified (diff)

comment:33 Changed 23 months ago by gh-tobiasdiez

There is also https://pypi.org/project/pipenv-to-requirements/ to generate the requirements file from a pipfile.

comment:34 Changed 23 months ago by mkoeppe

  • Dependencies changed from #29950 to #30719

comment:35 Changed 21 months ago by mkoeppe

  • Authors Matthias Koeppe deleted
  • Branch u/mkoeppe/at___bootstrap_time__generate_src_requirements_txt__src_constraints_txt__src_setup_cfg__install_requires__from_build_pkgs deleted
  • Commit 13088e302e305761994c04dca1d7b7d9d4b2f06d deleted
  • Dependencies changed from #30719 to #30719, #30865
  • Description modified (diff)
  • Summary changed from at ./bootstrap time, generate requirements.txt, constraints.txt, setup.cfg [install_requires], Pipfile from build/pkgs to sage_bootstrap: Add command "sage -package get-system-packages", supporting systems "install-requires", "requirements", "pipfile"
  • Work issues redo on top of #29950 deleted

comment:36 Changed 21 months ago by mkoeppe

  • Description modified (diff)

comment:37 Changed 21 months ago by mkoeppe

  • Keywords sd111 added

Hoping we can make progress on this ticket this week - https://wiki.sagemath.org/days111

comment:38 Changed 20 months ago by mkoeppe

  • Cc slabbe added
  • Description modified (diff)
  • Summary changed from sage_bootstrap: Add command "sage -package get-system-packages", supporting systems "install-requires", "requirements", "pipfile" to sage_bootstrap: Add command "sage -package list --output={install-requires,requirements,pipfile,debian,....}"

comment:39 Changed 20 months ago by mkoeppe

Rewriting tools used in the bootstrap phase in Python would also be useful for developing on Windows

comment:40 Changed 20 months ago by dimpase

hmm, what is so problematic about Windows here? Even if one talks about native, non-cygwin, it is still perfectly possible to run bash etc.

comment:41 Changed 20 months ago by mkoeppe

  • Description modified (diff)

comment:42 Changed 18 months ago by mkoeppe

  • Milestone changed from sage-9.3 to sage-9.4

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

comment:43 Changed 14 months ago by mkoeppe

  • Dependencies #30719, #30865 deleted

comment:44 Changed 13 months ago by mkoeppe

  • Milestone changed from sage-9.4 to sage-9.5

Setting a new milestone for this ticket based on a cursory review.

comment:45 Changed 8 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:46 Changed 5 months ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-9.7
Note: See TracTickets for help on using tickets.