Opened 21 months ago

Closed 21 months ago

Last modified 21 months ago

#29273 closed defect (fixed)

add system packages info for R and others

Reported by: dimpase Owned by:
Priority: major Milestone: sage-9.1
Component: build Keywords:
Cc: mkoeppe, fbissey, mjo Merged in:
Authors: Dima Pasechnik Reviewers: Matthias Koeppe
Report Upstream: N/A Work issues:
Branch: 0f704d0 (Commits, GitHub, GitLab) Commit:
Dependencies: #29129 Stopgaps:

Status badges

Description (last modified by dimpase)

this is a followup to #29053 which missed packages mentioned on #28884, and also some gentoo, arch, fedora, debian packages

Change History (50)

comment:1 Changed 21 months ago by mkoeppe

See also #29129

comment:2 Changed 21 months ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to u/dimpase/packages/r_packages_in_distros
  • Commit set to 9e0acb5cbf82aa92ffebd0041625359692d915a0
  • Dependencies set to #29129
  • Status changed from new to needs_review

New commits:

0fc301cAdd build/pkgs/{libgd|m4ri|ninja_build|perl_term_readline_gnu|yasm}/distros/*.txt - from repology
f7785cebuild/pkgs/fedora.txt: Add diffutils
6d2bbd8build/pkgs/arch.txt: Add 'which'
0aa90c1build/pkgs/debian.txt: Add ca-certificates
9e0acb5data for R packages

comment:3 Changed 21 months ago by git

  • Commit changed from 9e0acb5cbf82aa92ffebd0041625359692d915a0 to 9ebcd33c3d3dbb6408216ed25ebbb494512c7d4a

Branch pushed to git repo; I updated commit sha1. New commits:

9ebcd33gentoo R pkg

comment:4 Changed 21 months ago by git

  • Commit changed from 9ebcd33c3d3dbb6408216ed25ebbb494512c7d4a to e22f9745d735edb4075c9aaa046bec09b6adc684

Branch pushed to git repo; I updated commit sha1. New commits:

e22f974libgd package in gentoo

comment:5 Changed 21 months ago by git

  • Commit changed from e22f9745d735edb4075c9aaa046bec09b6adc684 to 8de5981c31c0c2e7feda82be6e8445f774a6be88

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8de5981libgd and ninja packages in gentoo

comment:6 Changed 21 months ago by git

  • Commit changed from 8de5981c31c0c2e7feda82be6e8445f774a6be88 to 9eff18be67d6afcc7349d6887e2cc3a18c6bad23

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

9eff18blibgd, symmetrica, ninja packages in gentoo

comment:7 Changed 21 months ago by dimpase

  • Description modified (diff)
  • Summary changed from add system packages info for R to add system packages info for R and others

comment:8 Changed 21 months ago by git

  • Commit changed from 9eff18be67d6afcc7349d6887e2cc3a18c6bad23 to fe0851f1093b64c8cbca835f018b205daf5fad11

Branch pushed to git repo; I updated commit sha1. New commits:

fe0851fr in conda

comment:9 Changed 21 months ago by git

  • Commit changed from fe0851f1093b64c8cbca835f018b205daf5fad11 to 7e129ef4f84577723b46544677b62869a3534e4a

Branch pushed to git repo; I updated commit sha1. New commits:

7e129efgentoo pkgs for givaro and pari

comment:10 Changed 21 months ago by mkoeppe

  • Cc fbissey added

comment:11 Changed 21 months ago by dimpase

  • Cc mjo added

comment:12 Changed 21 months ago by git

  • Commit changed from 7e129ef4f84577723b46544677b62869a3534e4a to 095e0bf498453b35dbf7eeb20f0be7e6f08f11fc

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

3fcf987Add build/pkgs/{libgd|m4ri|ninja_build|perl_term_readline_gnu|yasm}/distros/*.txt - from repology
d923e1fbuild/pkgs/fedora.txt: Add diffutils
7f4f7cdbuild/pkgs/arch.txt: Add 'which'
fa64cdbbuild/pkgs/debian.txt: Add ca-certificates
d8e3a13data for R packages
8dacd22gentoo R pkg
0ed2ccelibgd, symmetrica, ninja packages in gentoo
82d0896r in conda
74b4d8egentoo pkgs for givaro and pari
095e0bfpkgs for planarity and tachyon on debian and gentoo

comment:13 Changed 21 months ago by dimpase

rebased over 9.1.beta6

comment:14 Changed 21 months ago by fbissey

I am getting concerned that a rather unusual feature of gentoo may need additional description. The useflags. While packages in binary distro are built with all possible options [and when that is not possible either the best sets or variants of the package are offered] in Gentoo we leave the end user to choose a number of options by themselves. So not only do I want a package list, I want the option to go with it. Like for pari, we really want to use the gmp option (although I think it would just be slower in that case) and the doc option [not negotiable I even spent time making sure the doc was usable for cypari and sage when when it was part of it].

A list of useflags for Gentoo is at https://github.com/cschwan/sage-on-gentoo/blob/master/package.use/sage - some of the packages - pplpy, pynac and lcalc - are not in the main gentoo tree, only in the sage-on-gentoo overlay.

comment:15 follow-up: Changed 21 months ago by dimpase

perhaps there should be an extra optional datapoint for each gentoo package, the needed useflags, and then the installation advice would be generated by combining these flags into one list somehow?

comment:16 Changed 21 months ago by dimpase

by the way, could you people, fbissey or mjo, update sci-mathematics/gfan to 0.6.2? It's currently only 0.5, too old, in Gentoo.

I've filed https://bugs.gentoo.org/711492 to track this.

Last edited 21 months ago by dimpase (previous) (diff)

comment:17 Changed 21 months ago by git

  • Commit changed from 095e0bf498453b35dbf7eeb20f0be7e6f08f11fc to f8f9719b2d1d1874294f592a5a497c101c5afd85

Branch pushed to git repo; I updated commit sha1. New commits:

d86b532gp2c added, arb on gentoo
f8f9719remaining gentoo pkgs

comment:18 in reply to: ↑ 15 Changed 21 months ago by mkoeppe

Replying to dimpase:

perhaps there should be an extra optional datapoint for each gentoo package, the needed useflags, and then the installation advice would be generated by combining these flags into one list somehow?

Sounds good. Same format as a "package.use" file?

comment:19 Changed 21 months ago by dimpase

I've also filed https://bugs.gentoo.org/711498 to request m4rie to be packaged in Gentoo.

comment:20 Changed 21 months ago by git

  • Commit changed from f8f9719b2d1d1874294f592a5a497c101c5afd85 to a177aa7a3542b4bb4f32c8404e94110b703163a5

Branch pushed to git repo; I updated commit sha1. New commits:

a177aa7more for flint, glpk, mp*, gmp, ntl, sqlite

comment:21 follow-ups: Changed 21 months ago by mjo

The dependency syntax for Gentoo includes the USE flags in square brackets after the package name, like media-libs/gd[jpeg,png]. I noticed that we're going to need that information, but I'm not sure where/how these text files are meant to be displayed to the end user, so I'm not sure of "dependency" format or package.use format is better.

One idea I had was that, until all of sage is packaged, maybe we could have people create a local package set called @sagemath:

https://wiki.gentoo.org/wiki/Package_sets

If gentoo.txt contains entries like media-libs/gd[jpeg,png], then we could tell users to cat build/pkgs/*/distros/gentoo.txt > /etc/portage/sets/sagemath and then emerge --autounmask-use --autounmask-write @sagemath would automatically add those flags to package.use before installing the associated packages. That lets us use dependency syntax, and portage takes care of turning it into package.use entries.

That's how I'm keeping track of my system-sage packages for now. Package sets are a bad substitute for a meta package, but they're easier for non-technical users to create and manage.

OTOH, it may be simpler with package.use format to just specify a list of USE flags that works. Saying "we need this and this and this... you figure it out" is elegant, but experience has taught me that sometimes you just need to be able to tell the computer to "shut up and use this."

comment:22 in reply to: ↑ 21 Changed 21 months ago by mkoeppe

Replying to mjo:

I noticed that we're going to need that information, but I'm not sure where/how these text files are meant to be displayed to the end user, so I'm not sure of "dependency" format or package.use format is better.

Ideally, we will provide a script that formats a command line that the user can run directly. Similar to the apt-get lines shown in the Sage installation manual (formatted automatically as of #26964). See also #27351 (At end of configure, suggest a list of system packages to install) -- which should ideally also format a command line.

[...] Saying "we need this and this and this... you figure it out" is elegant, but experience has taught me that sometimes you just need to be able to tell the computer to "shut up and use this."

Indeed.

comment:23 in reply to: ↑ 21 Changed 21 months ago by mkoeppe

Replying to mjo:

The dependency syntax for Gentoo includes the USE flags in square brackets after the package name, like media-libs/gd[jpeg,png].

Sounds like this would be a better match for our gentoo.txt files than the package.use format.

comment:24 follow-up: Changed 21 months ago by dimpase

Does this syntax allow for things like foo/bar[baz/oz, buz/biz], i.e. with dependencies names spelled in full ?

There could also be non-dependency USE flags needed for configuration, but if the latter works it hopefully covers most of our needs.

comment:25 in reply to: ↑ 24 Changed 21 months ago by mjo

Replying to dimpase:

Does this syntax allow for things like foo/bar[baz/oz, buz/biz], i.e. with dependencies names spelled in full ?

There could also be non-dependency USE flags needed for configuration, but if the latter works it hopefully covers most of our needs.

The things inside the square brackets are the names of USE flags, not packages. So foo/bar[baz,buz] says that we need the package foo/bar built with both USE=baz and USE=buz set. Sometimes the USE flag names match up with package names, but ideally (and in general) they won't.

comment:26 Changed 21 months ago by dimpase

Let's handle the fine points of Gentoo setup on a follow-up ticket.

comment:27 follow-up: Changed 21 months ago by mkoeppe

I don't know how to check any of the gentoo things (see #29105).

comment:28 follow-up: Changed 21 months ago by dimpase

I am checking Gentoo things. The problem is that Gentoo is very fluid.

comment:29 in reply to: ↑ 28 Changed 21 months ago by mkoeppe

Replying to dimpase:

The problem is that Gentoo is very fluid.

Exactly because of that I want to define an environment using Docker so that the statement "it works on gentoo" is well-defined.

comment:30 in reply to: ↑ 27 Changed 21 months ago by dimpase

Replying to mkoeppe:

I don't know how to check any of the gentoo things (see #29105).

get the list of packages from all these gentoo.txt files, and run, as root,

emerge --ask <this list>

e.g. emerge --ask dev-lang/python foo/bar baz

this is interactive, and you either will get an error, or a prompt inviting you to start the installation. cf https://wiki.gentoo.org/wiki/Gentoo_Cheat_Sheet etc

(I don't have any machines ready to run docker, kernel options aren't right; I'll try on lxc using one of images here https://us.images.linuxcontainers.org/)

Last edited 21 months ago by dimpase (previous) (diff)

comment:31 follow-up: Changed 21 months ago by dimpase

An alternative would be to merge this as is (checking non-gentoo things), and deal with gentoo on a separate ticket, which might take a while.

comment:32 Changed 21 months ago by git

  • Commit changed from a177aa7a3542b4bb4f32c8404e94110b703163a5 to 7c9409836035b4505c747aa7559174e6ddc122f1

Branch pushed to git repo; I updated commit sha1. New commits:

7c94098added ntl flag to flint@gentoo, fixed typo

comment:33 follow-up: Changed 21 months ago by dimpase

In addition to the fix in the last commit, on the lxc gentoo/current image I picked on https://uk.images.linuxcontainers.org/, I needed to add

sci-mathematics/arb
sci-mathematics/flint
sci-mathematics/cliquer
sci-libs/m4ri
sci-mathematics/planarity
media-gfx/tachyon

to /etc/portage/package.keywords, as these packages are masked with ~amd64. Perhaps there is a quicker way to accomplish this.

givaro needed agreeing to a license treatment, i.e.

# required by sci-libs/givaro (argument)
>=sci-libs/givaro-4.1.1 CeCILL-B

added to /etc/portage/package.license. And, naturally, etc-update had to be run a number of times, resulting in /etc/portage/package.use/zz-autounmask getting

# package.use# required by sci-mathematics/flint[ntl] (argument)
>=sci-mathematics/flint-2.5.2-r1 ntl
# required by dev-lang/R-3.4.1::gentoo
# required by dev-lang/R (argument)
>=sys-libs/zlib-1.2.11-r2 minizip
# required by m4ri[png] (argument)
>=sci-libs/m4ri-20200115 png

after this, the build with all the gentoo.txt-mentioned packages started.

Thus, perhaps, installation instructions can have commands like run, as root:

cat $SAGE_ROOT/build/portage/package.keywords >> /etc/portage/package.keywords
Last edited 21 months ago by dimpase (previous) (diff)

comment:34 Changed 21 months ago by mkoeppe

  • Reviewers set to Matthias Koeppe
  • Status changed from needs_review to positive_review

comment:35 in reply to: ↑ 31 Changed 21 months ago by mkoeppe

Replying to dimpase:

deal with gentoo on a separate ticket, which might take a while.

Let's use #29105 (Extend dockerfile generator to gentoo) for that.

comment:36 in reply to: ↑ 33 Changed 21 months ago by mkoeppe

Replying to dimpase:

... Thus, perhaps, installation instructions can have commands like run, as root:

cat $SAGE_ROOT/build/portage/package.keywords >> /etc/portage/package.keywords

Sounds OK to me.

comment:37 Changed 21 months ago by git

  • Commit changed from 7c9409836035b4505c747aa7559174e6ddc122f1 to 8ae9764d5a4a1af452498c6eccfb3fb0bd3b1a51
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

8ae9764gsl and openblas libs

comment:38 Changed 21 months ago by dimpase

  • Status changed from needs_review to positive_review

The gentoo.txt files still need work, as the current configuration pulls in wrong blas/lapack, and llvm/clang (sic!), something we definitely don't want.

comment:39 Changed 21 months ago by dimpase

  • Description modified (diff)

comment:40 Changed 21 months ago by git

  • Commit changed from 8ae9764d5a4a1af452498c6eccfb3fb0bd3b1a51 to 31030557a5b692d3b1761b64ea1fc4874b3ea654
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

3103055m4ri must use png on gentoo

comment:41 Changed 21 months ago by dimpase

  • Status changed from needs_review to positive_review

OK, I should stop fiddling with this. Gentoo stuff needs moar workz.

comment:42 Changed 21 months ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:43 Changed 21 months ago by mkoeppe

  • Branch changed from u/dimpase/packages/r_packages_in_distros to u/mkoeppe/packages/r_packages_in_distros

comment:44 Changed 21 months ago by git

  • Commit changed from 31030557a5b692d3b1761b64ea1fc4874b3ea654 to 1b48aa12f5f1566022f67eb0ace593ce62769fe8

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

8ae9764gsl and openblas libs
3103055m4ri must use png on gentoo
1b48aa1Merge tag '9.1.beta7' into t/29273/packages/r_packages_in_distros

comment:45 Changed 21 months ago by mkoeppe

  • Status changed from needs_work to positive_review

comment:46 Changed 21 months ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:47 Changed 21 months ago by git

  • Commit changed from 1b48aa12f5f1566022f67eb0ace593ce62769fe8 to 0f704d03e5aaa1c6e34212f009bec4a50e2cbffa

Branch pushed to git repo; I updated commit sha1. New commits:

0fc301cAdd build/pkgs/{libgd|m4ri|ninja_build|perl_term_readline_gnu|yasm}/distros/*.txt - from repology
f7785cebuild/pkgs/fedora.txt: Add diffutils
6d2bbd8build/pkgs/arch.txt: Add 'which'
0aa90c1build/pkgs/debian.txt: Add ca-certificates
c49acdcMerge tag '9.1.beta7' into t/29129/add_more_system_packages_to_build_pkgs_spkg_distros_
0f704d0Merge branch 't/29129/add_more_system_packages_to_build_pkgs_spkg_distros_' into t/29273/packages/r_packages_in_distros

comment:48 Changed 21 months ago by mkoeppe

  • Status changed from needs_work to positive_review

comment:49 Changed 21 months ago by vbraun

  • Branch changed from u/mkoeppe/packages/r_packages_in_distros to 0f704d03e5aaa1c6e34212f009bec4a50e2cbffa
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:50 Changed 21 months ago by dimpase

  • Commit 0f704d03e5aaa1c6e34212f009bec4a50e2cbffa deleted

a followup on #29352

Note: See TracTickets for help on using tickets.