Add pip packages to conda environments, add missing conda.txt files

[Matthias Köppe]

We have a number of pip-installable optional packages, for which there is no conda package.

Here we add a mechanism to add such packages using a pip: line in src/environment-optional.yml and src/environment-dev.yml (#33740).

We also add lots of conda.txt files.

To test, follow the instructions in ​https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental using src/environment-optional.yml or src/environment-dev.yml

(A use case for the pip line in src/environment-dev.yml is #34524: Pip package git_trac_command.)

[Matthias Köppe]

(Without #33589) I also get

(sage-sh) gitpod@sagemath-sagetracmirror-72wwexpcfmb:sagetrac-mirror$ git trac checkout 33608
Loading ticket #33608...
Checking out Trac #33608 remote branch u/klee/33608 -> local branch t/33608/33608...
Traceback (most recent call last):
    STDERR: Warning: Permanently added the ECDSA host key for IP address '104.197.143.230' to the list of known hosts.
    STDERR: Load key "/home/gitpod/.ssh/id_rsa": invalid format
    STDERR: git@trac.sagemath.org: Permission denied (publickey).
    STDERR: fatal: Could not read from remote repository.
    STDERR: 
    STDERR: Please make sure you have the correct access rights
    STDERR: and the repository exists.

[Tobias Diez]

Did you followed the steps outlined in sage's gitpod docs to add the ssh key?

[Matthias Köppe]

No. In #33589 we should make sure that fetching can be done without these steps.

[Matthias Köppe]

Last 10 new commits:

​a5d10a2	Fix typo
​4950bcf	Add fallback to github mirror
​c235ead	Disable pushing to github
​40d1e23	Always disable pushing to origin
​89591c4	Remove wrong trac from prebuild
​3d450f3	Update docs
​e9035ae	Silence semi-expected errors during (pre)build
​8432e1d	Merge #33589
​8aaaf07	build/pkgs/python3/distros/debian.txt: New
​eacd393	docker/.gitpod.Dockerfile: Install pip for system python, install git-trac-command

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3f71861

docker/.gitpod.Dockerfile: hardcode python3-pip

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​866a833

docker/.gitpod.Dockerfile: Run apt-get update first

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

​ccbc21a	bootstrap, bootstrap-conda, src/doc/bootstrap: Use build/bin/sage-package directly
​4560895	src/doc/Makefile: Remove explicit use of SAGE_ROOT and the top-level sage script
​7759f3c	Generate en/reference/repl/options.txt at build time, not bootstrap time
​58ab878	bootstrap-conda: Fix typo, generate->generating to match messages from bootstrap
​12ea8ce	build/pkgs/_develop/distros: Add packages providing gpgconf
​238b7bb	build/pkgs/_develop/distros: Add packages providing ssh
​01a126c	Merge #33851
​a74ccaa	build/pkgs/git_trac_command: New
​e28f6d9	build/pkgs/_develop/dependencies: Add git_trac_command
​d2b8c4e	bootstrap-conda: Also generate pip: lines

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ffde371

bootstrap-conda: Do not include packages that declare SAGERUNTIME as a dependency

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d6a916c

Merge tag '9.7.beta1' into t/33613/gitpod__install_git_trac_command

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​67c8238	src/bin/sage: Print error to error output
​bc77bfd	bootstrap-conda: Write an additional comment
​4c2adf7	bootstrap-conda: Rewrite using fewer redirects
​ac2c48d	build/pkgs/git_trac_command: New
​4a0b6f4	bootstrap-conda: Also generate pip: lines
​ef01ba4	bootstrap-conda: Do not include packages that declare SAGERUNTIME as a dependency
​8e511f2	bootstrap-conda: Hardcode git_trac_command as a devel package

[Matthias Köppe]

Rebased away from #33822, #33851.

Ready for review

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​94d6ec7

Merge tag '9.7.beta3' into t/33613/gitpod__install_git_trac_command

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2019f47

bootstrap-conda: Also check dependencies_order_only

[Dima Pasechnik]

not quite sure what should be tested here.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d169385	34096: Fix typo: enviroment -> environment
​10f1e42	Merge #34096

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​facc5e2

build/pkgs/*/distros/conda.txt: Add some packages

[Tobias Diez]

Why not simply publish the trac git command package to conda forge? Moreover, given the limitations that conda has with pip-installed packages (especially concerning dependency resolution), I'm not a big fan of the semi-automatic pip fallback. What do you think about supporting the syntax pip: xyz in xyz/distros/conda.txt to signal that this package should be installed via pip. Finally, the spkg documentation needs to be changed for those pip-installed packages as well, since conda install xyz is not the correct way for them.

[Matthias Köppe]

Replying to gh-tobiasdiez:

Why not simply publish the trac git command package to conda forge?

It's not a good idea for a project to take on downstream packaging duties.

[Matthias Köppe]

Replying to gh-tobiasdiez:

the spkg documentation needs to be changed for those pip-installed packages as well, since conda install xyz is not the correct way for them.

conda install xyz is not issued for these packages because conda.txt is not present.

[Matthias Köppe]

Replying to gh-tobiasdiez:

given the limitations that conda has with pip-installed packages (especially concerning dependency resolution), I'm not a big fan of the semi-automatic pip fallback.

If there ever is an actual problem with a pip installed package, it's easy to override it with an empty conda.txt file.

[Matthias Köppe]

Replying to gh-tobiasdiez:

I'm not a big fan of the semi-automatic pip fallback

Being semi-automatic makes it require less maintenance.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​c65e4b0	Merge tag '9.7.beta7' into t/33613/gitpod__install_git_trac_command
​9d07598	build/pkgs/*/distros/conda.txt: Add some packages
​6e349d5	Merge #34263
​fc7eb33	34274: Add sirocco distro information: conda, fedora
​828fb13	Merge #34274

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a004e00

build/pkgs/python_build/distros/conda.txt: New

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d508d7b

build/pkgs: Add many more distros/conda.txt files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​098a8e7	build/pkgs: Add some more distros/conda.txt files
​c56aa71	bootstrap-conda: Also add pip lines when install-requires.txt is present

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c13eaf1	build/pkgs: Add even more distros/conda.txt files
​1c47261	bootstrap-conda: Do not include pip lines for packages for which we have source directories

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6d06720

build/pkgs: Add even more distros/conda.txt files (fixup)

[git]

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

​3f6660b	bootstrap-conda: Write an additional comment
​04045d3	build/pkgs/_develop/dependencies: Add pytest, pytest_xdist
​0856a00	bootstrap-conda: Write an additional comment (fixup)
​8f17c11	bootstrap-conda: Rewrite using fewer redirects
​15598c5	build/pkgs/_develop/distros: Add packages providing gpgconf
​eab7c56	build/pkgs/_develop/distros: Add packages providing ssh
​6592b38	README.md: All developers should install _bootstrap packages
​2e96172	src/doc/bootstrap: Generate *-develop.txt
​c96c7c5	src/doc/en/installation/source.rst: Include *-develop.txt and *-recommended.txt
​175a758	Merge #33851

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6dfc583

build/pkgs/_develop/dependencies: Add git_trac_command

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d90b64c

Revert "build/pkgs/git_trac_command: New"

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0e4a696

Merge tag '9.7.rc1' into t/33613/gitpod__install_git_trac_command

[Dima Pasechnik]

this should be merged soon.

[Matthias Köppe]

Thank you!