Opened 8 months ago

Closed 3 months ago

#33613 closed enhancement (fixed)

Add pip packages to conda environments, add missing conda.txt files

Reported by: Matthias Köppe Owned by:
Priority: major Milestone: sage-9.8
Component: porting Keywords:
Cc: Tobias Diez, Dima Pasechnik, Isuru Fernando, Julian Rüth, Samuel Lelièvre, Yuan Zhou Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 0e4a696 (Commits, GitHub, GitLab) Commit: 0e4a696fc3685afa56c2533e57b59fd765946072
Dependencies: Stopgaps:

Status badges

Description (last modified by Matthias Köppe)

We have a number of pip-installable optional packages, for which there is no conda package.

Here we add a mechanism to add such packages using a pip: line in src/environment-optional.yml and src/environment-dev.yml (#33740).

We also add lots of conda.txt files.

To test, follow the instructions in https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental using src/environment-optional.yml or src/environment-dev.yml

(A use case for the pip line in src/environment-dev.yml is #34524: Pip package git_trac_command.)

Change History (65)

comment:1 Changed 8 months ago by Matthias Köppe

Summary: gitpod: Install sage-trac-commandgitpod: Install git-trac-command

comment:2 Changed 8 months ago by Matthias Köppe

Description: modified (diff)

comment:3 Changed 8 months ago by Matthias Köppe

(Without #33589) I also get

(sage-sh) gitpod@sagemath-sagetracmirror-72wwexpcfmb:sagetrac-mirror$ git trac checkout 33608
Loading ticket #33608...
Checking out Trac #33608 remote branch u/klee/33608 -> local branch t/33608/33608...
Traceback (most recent call last):
    STDERR: Warning: Permanently added the ECDSA host key for IP address '104.197.143.230' to the list of known hosts.
    STDERR: Load key "/home/gitpod/.ssh/id_rsa": invalid format
    STDERR: git@trac.sagemath.org: Permission denied (publickey).
    STDERR: fatal: Could not read from remote repository.
    STDERR: 
    STDERR: Please make sure you have the correct access rights
    STDERR: and the repository exists.

comment:4 Changed 8 months ago by Tobias Diez

Did you followed the steps outlined in sage's gitpod docs to add the ssh key?

comment:5 Changed 8 months ago by Matthias Köppe

Dependencies: #33589

No. In #33589 we should make sure that fetching can be done without these steps.

comment:6 Changed 8 months ago by Matthias Köppe

Branch: u/mkoeppe/gitpod__install_git_trac_command

comment:7 Changed 8 months ago by Matthias Köppe

Authors: Matthias Koeppe
Commit: eacd393e820e277a5951c232d7ff1b28e63b7c87

Last 10 new commits:

a5d10a2Fix typo
4950bcfAdd fallback to github mirror
c235eadDisable pushing to github
40d1e23Always disable pushing to origin
89591c4Remove wrong trac from prebuild
3d450f3Update docs
e9035aeSilence semi-expected errors during (pre)build
8432e1dMerge #33589
8aaaf07build/pkgs/python3/distros/debian.txt: New
eacd393docker/.gitpod.Dockerfile: Install pip for system python, install git-trac-command

comment:8 Changed 8 months ago by git

Commit: eacd393e820e277a5951c232d7ff1b28e63b7c873f718616439635d213a2d50ac7263f0e6e549c3f

Branch pushed to git repo; I updated commit sha1. New commits:

3f71861docker/.gitpod.Dockerfile: hardcode python3-pip

comment:9 Changed 8 months ago by git

Commit: 3f718616439635d213a2d50ac7263f0e6e549c3f866a833c649f1388b2f552cddbe258eea96b8ecf

Branch pushed to git repo; I updated commit sha1. New commits:

866a833docker/.gitpod.Dockerfile: Run apt-get update first

comment:10 Changed 8 months ago by Matthias Köppe

Milestone: sage-9.6sage-9.7

comment:11 Changed 7 months ago by Matthias Köppe

Description: modified (diff)
Summary: gitpod: Install git-trac-commandScript package git_trac_command

comment:12 Changed 7 months ago by Matthias Köppe

Description: modified (diff)

comment:13 Changed 7 months ago by Matthias Köppe

Dependencies: #33589#33740, #33851

comment:14 Changed 7 months ago by Matthias Köppe

Dependencies: #33740, #33851#33740, #33822, #33851

comment:15 Changed 7 months ago by Matthias Köppe

Summary: Script package git_trac_commandPip package git_trac_command

comment:16 Changed 7 months ago by git

Commit: 866a833c649f1388b2f552cddbe258eea96b8ecfd2b8c4e2a4098d2bdecabb531e30feaa9b130306

Branch pushed to git repo; I updated commit sha1. This was a forced push. Last 10 new commits:

ccbc21abootstrap, bootstrap-conda, src/doc/bootstrap: Use build/bin/sage-package directly
4560895src/doc/Makefile: Remove explicit use of SAGE_ROOT and the top-level sage script
7759f3cGenerate en/reference/repl/options.txt at build time, not bootstrap time
58ab878bootstrap-conda: Fix typo, generate->generating to match messages from bootstrap
12ea8cebuild/pkgs/_develop/distros: Add packages providing gpgconf
238b7bbbuild/pkgs/_develop/distros: Add packages providing ssh
01a126cMerge #33851
a74ccaabuild/pkgs/git_trac_command: New
e28f6d9build/pkgs/_develop/dependencies: Add git_trac_command
d2b8c4ebootstrap-conda: Also generate pip: lines

comment:17 Changed 7 months ago by git

Commit: d2b8c4e2a4098d2bdecabb531e30feaa9b130306ffde3710e644845ed19793054379f4a6dd6d5f68

Branch pushed to git repo; I updated commit sha1. New commits:

ffde371bootstrap-conda: Do not include packages that declare SAGERUNTIME as a dependency

comment:18 Changed 7 months ago by Matthias Köppe

Description: modified (diff)
Status: newneeds_review

comment:19 Changed 6 months ago by git

Commit: ffde3710e644845ed19793054379f4a6dd6d5f68d6a916c084e0a34af856f83a2d5964193f9fb421

Branch pushed to git repo; I updated commit sha1. New commits:

d6a916cMerge tag '9.7.beta1' into t/33613/gitpod__install_git_trac_command

comment:20 Changed 6 months ago by Matthias Köppe

Cc: Dima Pasechnik added

comment:21 Changed 6 months ago by Matthias Köppe

Summary: Pip package git_trac_commandPip package git_trac_command, add pip packages to conda environments

comment:22 Changed 6 months ago by Matthias Köppe

Dependencies: #33740, #33822, #33851#33852

comment:23 Changed 6 months ago by git

Commit: d6a916c084e0a34af856f83a2d5964193f9fb4218e511f232585f98f18befe40945be71964b21d41

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

67c8238src/bin/sage: Print error to error output
bc77bfdbootstrap-conda: Write an additional comment
4c2adf7bootstrap-conda: Rewrite using fewer redirects
ac2c48dbuild/pkgs/git_trac_command: New
4a0b6f4bootstrap-conda: Also generate pip: lines
ef01ba4bootstrap-conda: Do not include packages that declare SAGERUNTIME as a dependency
8e511f2bootstrap-conda: Hardcode git_trac_command as a devel package

comment:24 Changed 6 months ago by Matthias Köppe

Rebased away from #33822, #33851.

Ready for review

comment:25 Changed 5 months ago by git

Commit: 8e511f232585f98f18befe40945be71964b21d4194d6ec769cabfc91c18649447a67584a22ab111d

Branch pushed to git repo; I updated commit sha1. New commits:

94d6ec7Merge tag '9.7.beta3' into t/33613/gitpod__install_git_trac_command

comment:26 Changed 5 months ago by git

Commit: 94d6ec769cabfc91c18649447a67584a22ab111d2019f4791f0bc6edf6c9f35a1ae618206d02623b

Branch pushed to git repo; I updated commit sha1. New commits:

2019f47bootstrap-conda: Also check dependencies_order_only

comment:27 Changed 5 months ago by Dima Pasechnik

not quite sure what should be tested here.

comment:28 Changed 5 months ago by Matthias Köppe

Dependencies: #33852#33852, #34096

comment:29 Changed 5 months ago by git

Commit: 2019f4791f0bc6edf6c9f35a1ae618206d02623b10f1e420a0f2ad2d6acafd0b390f60cfd9e7a7bb

Branch pushed to git repo; I updated commit sha1. New commits:

d16938534096: Fix typo: enviroment -> environment
10f1e42Merge #34096

comment:30 Changed 5 months ago by Matthias Köppe

Description: modified (diff)

comment:31 Changed 5 months ago by git

Commit: 10f1e420a0f2ad2d6acafd0b390f60cfd9e7a7bbfacc5e2b890ab3f6925552a7f569d022d49d1461

Branch pushed to git repo; I updated commit sha1. New commits:

facc5e2build/pkgs/*/distros/conda.txt: Add some packages

comment:32 Changed 5 months ago by Tobias Diez

Status: needs_reviewneeds_work

Why not simply publish the trac git command package to conda forge? Moreover, given the limitations that conda has with pip-installed packages (especially concerning dependency resolution), I'm not a big fan of the semi-automatic pip fallback. What do you think about supporting the syntax pip: xyz in xyz/distros/conda.txt to signal that this package should be installed via pip. Finally, the spkg documentation needs to be changed for those pip-installed packages as well, since conda install xyz is not the correct way for them.

comment:33 in reply to:  32 Changed 5 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

Why not simply publish the trac git command package to conda forge?

It's not a good idea for a project to take on downstream packaging duties.

comment:34 in reply to:  32 Changed 5 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

the spkg documentation needs to be changed for those pip-installed packages as well, since conda install xyz is not the correct way for them.

conda install xyz is not issued for these packages because conda.txt is not present.

comment:35 in reply to:  32 Changed 5 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

given the limitations that conda has with pip-installed packages (especially concerning dependency resolution), I'm not a big fan of the semi-automatic pip fallback.

If there ever is an actual problem with a pip installed package, it's easy to override it with an empty conda.txt file.

comment:36 Changed 5 months ago by Matthias Köppe

Status: needs_workneeds_review

comment:37 in reply to:  32 Changed 5 months ago by Matthias Köppe

Replying to gh-tobiasdiez:

I'm not a big fan of the semi-automatic pip fallback

Being semi-automatic makes it require less maintenance.

comment:38 Changed 5 months ago by Matthias Köppe

Cc: Isuru Fernando added

comment:39 Changed 4 months ago by Matthias Köppe

Cc: Julian Rüth added

comment:40 Changed 4 months ago by Matthias Köppe

Dependencies: #33852, #34096#33852, #34096, #34263

comment:41 Changed 4 months ago by Matthias Köppe

Dependencies: #33852, #34096, #34263#33852, #34096, #34263, #34274

comment:42 Changed 4 months ago by git

Commit: facc5e2b890ab3f6925552a7f569d022d49d1461828fb13d2d4e6ee8b1ab226fdbcc8b2f90f91e2a

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c65e4b0Merge tag '9.7.beta7' into t/33613/gitpod__install_git_trac_command
9d07598build/pkgs/*/distros/conda.txt: Add some packages
6e349d5Merge #34263
fc7eb3334274: Add sirocco distro information: conda, fedora
828fb13Merge #34274

comment:43 Changed 4 months ago by Matthias Köppe

Dependencies: #33852, #34096, #34263, #34274#34263, #34274

comment:44 Changed 4 months ago by git

Commit: 828fb13d2d4e6ee8b1ab226fdbcc8b2f90f91e2aa004e002d8353e08e5d1f7d6601ac2d434c50f30

Branch pushed to git repo; I updated commit sha1. New commits:

a004e00build/pkgs/python_build/distros/conda.txt: New

comment:45 Changed 4 months ago by git

Commit: a004e002d8353e08e5d1f7d6601ac2d434c50f30d508d7bff7e4d2fce3e097b58ba4726aa973b65d

Branch pushed to git repo; I updated commit sha1. New commits:

d508d7bbuild/pkgs: Add many more distros/conda.txt files

comment:46 Changed 4 months ago by git

Commit: d508d7bff7e4d2fce3e097b58ba4726aa973b65dc56aa71a7c48b63754554f9449152197123f6c1c

Branch pushed to git repo; I updated commit sha1. New commits:

098a8e7build/pkgs: Add some more distros/conda.txt files
c56aa71bootstrap-conda: Also add pip lines when install-requires.txt is present

comment:47 Changed 4 months ago by git

Commit: c56aa71a7c48b63754554f9449152197123f6c1c1c47261840fcee978b7471ff648a2b39eecde5fa

Branch pushed to git repo; I updated commit sha1. New commits:

c13eaf1build/pkgs: Add even more distros/conda.txt files
1c47261bootstrap-conda: Do not include pip lines for packages for which we have source directories

comment:48 Changed 4 months ago by git

Commit: 1c47261840fcee978b7471ff648a2b39eecde5fa6d067200c54e48597c262695b866cc3c321ff766

Branch pushed to git repo; I updated commit sha1. New commits:

6d06720build/pkgs: Add even more distros/conda.txt files (fixup)

comment:49 Changed 4 months ago by Matthias Köppe

Priority: minormajor

comment:50 Changed 4 months ago by Matthias Köppe

Summary: Pip package git_trac_command, add pip packages to conda environmentsPip package git_trac_command, add pip packages to conda environments, add missing conda.txt files

comment:51 Changed 4 months ago by Matthias Köppe

Cc: Samuel Lelièvre added

comment:52 Changed 3 months ago by Matthias Köppe

Dependencies: #34263, #34274#34263, #34274, #33851

comment:53 Changed 3 months ago by git

Commit: 6d067200c54e48597c262695b866cc3c321ff766175a7589a3fed304ac483e56f177be2b30b557ec

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

3f6660bbootstrap-conda: Write an additional comment
04045d3build/pkgs/_develop/dependencies: Add pytest, pytest_xdist
0856a00bootstrap-conda: Write an additional comment (fixup)
8f17c11bootstrap-conda: Rewrite using fewer redirects
15598c5build/pkgs/_develop/distros: Add packages providing gpgconf
eab7c56build/pkgs/_develop/distros: Add packages providing ssh
6592b38README.md: All developers should install _bootstrap packages
2e96172src/doc/bootstrap: Generate *-develop.txt
c96c7c5src/doc/en/installation/source.rst: Include *-develop.txt and *-recommended.txt
175a758Merge #33851

comment:54 Changed 3 months ago by git

Commit: 175a7589a3fed304ac483e56f177be2b30b557ec6dfc5832867683f2b9a74de39f72ee2d0c0f3101

Branch pushed to git repo; I updated commit sha1. New commits:

6dfc583build/pkgs/_develop/dependencies: Add git_trac_command

comment:55 Changed 3 months ago by Matthias Köppe

Cc: Yuan Zhou added

comment:56 Changed 3 months ago by git

Commit: 6dfc5832867683f2b9a74de39f72ee2d0c0f3101d90b64c6b7957b6c2294a1f0efb0447fcce138fa

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d90b64cRevert "build/pkgs/git_trac_command: New"

comment:57 Changed 3 months ago by Matthias Köppe

Description: modified (diff)
Summary: Pip package git_trac_command, add pip packages to conda environments, add missing conda.txt filesAdd pip packages to conda environments, add missing conda.txt files

comment:58 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:59 Changed 3 months ago by git

Commit: d90b64c6b7957b6c2294a1f0efb0447fcce138fa0e4a696fc3685afa56c2533e57b59fd765946072

Branch pushed to git repo; I updated commit sha1. New commits:

0e4a696Merge tag '9.7.rc1' into t/33613/gitpod__install_git_trac_command

comment:60 Changed 3 months ago by Matthias Köppe

Dependencies: #34263, #34274, #33851

comment:61 Changed 3 months ago by Matthias Köppe

Description: modified (diff)

comment:62 Changed 3 months ago by Dima Pasechnik

Reviewers: Dima Pasechnik
Status: needs_reviewpositive_review

this should be merged soon.

comment:63 Changed 3 months ago by Matthias Köppe

Thank you!

comment:64 Changed 3 months ago by Matthias Köppe

Milestone: sage-9.7sage-9.8

comment:65 Changed 3 months ago by Volker Braun

Branch: u/mkoeppe/gitpod__install_git_trac_command0e4a696fc3685afa56c2533e57b59fd765946072
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.