Opened 8 months ago

Last modified 2 months ago

#32753 needs_review enhancement

Prefer Linux package manager over linuxbrew; add nix detection

Reported by: gh-tobiasdiez Owned by:
Priority: major Milestone: sage-9.7
Component: build: configure Keywords:
Cc: mkoeppe, gh-collares, jipilab Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: N/A Work issues:
Branch: u/mkoeppe/prefer_linux_package_manager_over_linuxbrew (Commits, GitHub, GitLab) Commit: 9af33a1b8678fc351413a1fa67daceb7b9439161
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

If brew is installed on Linux, Sagemath thinks this is the primary package manager and, e.g., configure suggests to run brew ... to install new packages. However, linuxbrew is untested as a platform for Sage, and so this is not a good idea.

We change build/bin/sage-guess-package-system so that brew is only tested after the Linux package managers.

We also add a detection for nix-env.

Change History (24)

comment:1 Changed 6 months ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6

comment:2 Changed 5 months ago by dimpase

I now see this in our gitpod.

comment:3 Changed 5 months ago by mkoeppe

  • Branch set to u/mkoeppe/prefer_linux_package_manager_over_linuxbrew

comment:4 Changed 5 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Cc gh-collares added
  • Commit set to 90f7ced00b74cca051c167c271a9a49155ad0710
  • Description modified (diff)
  • Status changed from new to needs_review
  • Summary changed from Prefer Linux package manager over linuxbrew to Prefer Linux package manager over linuxbrew; add nix detection

For nix, do we need to check whether a nix environment is active, like we do for conda?


New commits:

04f9749build/bin/sage-guess-package-system: check linuxbrew after linux package managers because it is untested
90f7cedbuild/bin/sage-guess-package-system: Check for nix

comment:5 follow-ups: Changed 5 months ago by gh-collares

To be honest with you, I have never used nix-env (the manual offers a declarative path and a nix-env path, and strongly suggests nix-env is the worst of the two alternatives). I think supporting nix-env is valid, but given my lack of experience with it I am not sure what the workflow would be in this case. You can certainly install packages with nix-env in your "default profile", and the executables would be in PATH without having to activate an environment.

People who follow the declarative approach and want to build Sage from source would bypass nix-env and write a derivation (a declarative recipe, basically) to pull in the required dependencies, and then pass this to nix-shell to obtain an ephemeral virtualenv-like environment with those deps available. Or, instead of writing the derivation themselves, they could also use Nixpkgs's Sage derivation to get the same build environment we use. Either way, they would be able to build Sage from Git sources using Nix packages for the dependencies, but nix-env would be none the wiser.

It would be useful to know a few things so I can understand how this fits into the Nix model: Is it just for printing a command to install dependencies? Or does it query installed packages as well? What happens if Singular (say) just exists on PATH but nix-env doesn't report it as installed? Pointing me to the relevant source file would be enough, no need to answer in detail. Thanks!

comment:6 in reply to: ↑ 5 Changed 5 months ago by mkoeppe

Replying to gh-collares:

People who follow the declarative approach and want to build Sage from source would bypass nix-env and write a derivation (a declarative recipe, basically) to pull in the required dependencies, and then pass this to nix-shell to obtain an ephemeral virtualenv-like environment with those deps available. Or, instead of writing the derivation themselves, they could also use Nixpkgs's Sage derivation to get the same build environment we use.

It would be great to have instructions how to do Sage development using nixpkgs in our installation manual.

Other than figuring out how to provision a nix environment with packages on top of the nixos/nix Docker image (so I can say tox -e docker-nixos-standard), I have no experience with nix.

By the way, I am getting the error:

Step 6/46 : RUN  nix-env --install  gd iml m4ri gfan singular mpfi cmake symmetrica flintqs freetype cddlib libbraiding brial ppl sympow zn_poly readline R sqlite tachyon giac m4rie blas lapack ntl glpk fflas-ffpack libmpc gsl boost nauty binutils gnumake gnum4 perl python3 gnutar bc gcc bash flint arb pari cliquer libhomfly openssl gengetopt planarity lrcalc palp ecm givaro ecl eclib lcalc rankwidth qhull gettext autoconf automake libtool pkg-config
 ---> Running in 0dcbda1f2003
error: selector 'libhomfly' matches no derivations

so it may be that some work on the nix.txt files is needed.

comment:7 in reply to: ↑ 5 Changed 5 months ago by mkoeppe

Replying to gh-collares:

It would be useful to know a few things so I can understand how this fits into the Nix model: Is it just for printing a command to install dependencies?

Yes, it's for the system package advice that is printed at the end of a configure run.

The code for this is in build/bin/sage-print-system-package-command.

comment:8 in reply to: ↑ 5 Changed 5 months ago by mkoeppe

Replying to gh-collares:

What happens if Singular (say) just exists on PATH

configure only checks what is available in PATH.

For packages where we don't find a system package in this way, if there is system package information in build/pkgs/*/nix.txt for the package, we issue the system package advice. No package manager queries are done -- this is all left to the user.

(This is a design principle for this code -- it does not attempt to take care of system package installation tasks for the user; instead it only informs/educates the user about possible commands.)

comment:9 follow-up: Changed 5 months ago by gh-collares

Thanks for the information, this is very helpful! So, yeah, I think recommending nix-env is fine, since people who use the declarative approach can just copy and paste the list of packages into their derivations.

As for the libhomfly error: The preferred way to install packages using nix-env is nix-env -f '<nixpkgs>' -iA m4ri libhomfly (say). The -A flag specifies a saner way of finding the relevant package; in this case, the -A-less version caught a typo in the Nix package. I've submitted a PR to fix it in Nixpkgs, but ideally the nix-env invocation would be switched to something like the command I mentioned above.

I will try to write some docs for Sage development with nixpkgs as part of the Sage 9.6 release.

comment:10 Changed 5 months ago by git

  • Commit changed from 90f7ced00b74cca051c167c271a9a49155ad0710 to e28862883054256363cae86d6c4dfaad0fb844ca

Branch pushed to git repo; I updated commit sha1. New commits:

e288628build/bin/sage-print-system-package-command (nix): Use nix-env --install --attr

comment:11 Changed 5 months ago by mkoeppe

Not sure what the -f '<nixpkgs>' is - is there something I need to add?

comment:12 in reply to: ↑ 9 Changed 5 months ago by mkoeppe

Replying to gh-collares:

The -A flag specifies a saner way of finding the relevant package

Thanks, this helped!

comment:13 Changed 5 months ago by mkoeppe

configure now gives the following advice:

checking for the package system in use... nix
configure:

    hint: installing the following system packages, if not
    already present, is recommended and may avoid having to
    build them (though some may have to be built anyway):

      $ nix-env --install --attr ecl eclib fflas-ffpack giac givaro gsl iml lcalc libhomfly blas lapack palp pari R

configure:

    hint: installing the following system packages, if not
    already present, may provide additional optional features:

      $ nix-env --install --attr ffmpeg graphviz imagemagick texinfo graphviz libxml2 lrs pdf2svg polymake

comment:14 Changed 5 months ago by gh-collares

This is perfect, many thanks! I don't know why I needed the -f <nixpkgs> here, perhaps because I am on NixOS? It's best to leave it out if it works outside NixOS.

comment:15 Changed 5 months ago by git

  • Commit changed from e28862883054256363cae86d6c4dfaad0fb844ca to 3d96dc560bd838feda56b26ccc48bf90cd641fd2

Branch pushed to git repo; I updated commit sha1. New commits:

3d96dc5build/bin/sage-print-system-package-command, build/bin/write-dockerfile.sh: Use nix-env -f '<nixpkgs>' --install --attr

comment:16 Changed 5 months ago by mkoeppe

OK, looks like I do need this switch

comment:17 Changed 5 months ago by git

  • Commit changed from 3d96dc560bd838feda56b26ccc48bf90cd641fd2 to b07fece963b3a656f1a1be10635f8c24416edf04

Branch pushed to git repo; I updated commit sha1. New commits:

b07fecebuild/pkgs/gfortran/distros/nix.txt: New

comment:18 Changed 5 months ago by mkoeppe

Now I am getting

building '/nix/store/r2w1z0912rilh4l1yyg9jv3d66bnjk99-user-environment.drv'...
error: packages '/nix/store/9iry5bm9cxdbspap6qi835g67kvdninx-boost-1.77.0/lib/libboost_thread.so' and '/nix/store/6x1m911w8378qbxjf9pcli9hijrqx1km-nix-2.6.0/lib/libboost_thread.so' have the same priority 5; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)
error: builder for '/nix/store/r2w1z0912rilh4l1yyg9jv3d66bnjk99-user-environment.drv' failed with exit code 1
The command '/bin/sh -c nix-env -f '<nixpkgs>' --install --attr  gd iml m4ri gfan gfortran singular mpfi cmake symmetrica flintqs freetype cddlib libbraiding brial ppl sympow zn_poly readline R sqlite tachyon giac m4rie blas lapack ntl glpk fflas-ffpack libmpc gsl boost nauty binutils gnumake gnum4 perl python3 gnutar bc gcc bash flint arb pari cliquer libhomfly openssl gengetopt planarity lrcalc palp ecm givaro ecl eclib lcalc libatomic_ops rankwidth qhull gettext autoconf automake libtool pkg-config' returned a non-zero code: 100

comment:19 Changed 5 months ago by gh-collares

I've never seen this error and I am not exactly sure how to fix it. Can we just remove Boost's nix.txt, as a workaround? Hopefully there are no other conflicts like this.

comment:20 Changed 5 months ago by git

  • Commit changed from b07fece963b3a656f1a1be10635f8c24416edf04 to 9af33a1b8678fc351413a1fa67daceb7b9439161

Branch pushed to git repo; I updated commit sha1. New commits:

9af33a1build/pkgs/boost_cropped/distros/nix.txt: Remove

comment:21 Changed 5 months ago by mkoeppe

That worked, thanks. Next error:

building '/nix/store/4vmcjc3jqzvzsy1mvydsl6k5glbbvvvm-user-environment.drv'...
error: packages '/nix/store/cm5d8zx8jf24f6vjxzdx9by3k6rn5zd2-gfortran-wrapper-9.3.0-man/share/man/man7/fsf-funding.7.gz' and '/nix/store/310zm0mxwmk7ynsk0qjb7hiaf734s8p8-gcc-wrapper-10.3.0-man/share/man/man7/fsf-funding.7.gz' have the same priority 10; use 'nix-env --set-flag priority NUMBER INSTALLED_PKGNAME' to change the priority of one of the conflicting packages (0 being the highest priority)
error: builder for '/nix/store/4vmcjc3jqzvzsy1mvydsl6k5glbbvvvm-user-environment.drv' failed with exit code 1
The command '/bin/sh -c nix-env -f '<nixpkgs>' --install --attr  gd iml m4ri gfan gfortran singular mpfi cmake symmetrica flintqs freetype cddlib libbraiding brial ppl sympow zn_poly readline R sqlite tachyon giac m4rie blas lapack ntl glpk fflas-ffpack libmpc gsl nauty binutils gnumake gnum4 perl python3 gnutar bc gcc bash flint arb pari cliquer libhomfly openssl gengetopt planarity lrcalc palp ecm givaro ecl eclib lcalc libatomic_ops rankwidth qhull gettext autoconf automake libtool pkg-config' returned a non-zero code: 100

comment:22 Changed 5 months ago by mkoeppe

  • Status changed from needs_review to needs_info

comment:23 Changed 3 months ago by mkoeppe

  • Milestone changed from sage-9.6 to sage-9.7

comment:24 Changed 2 months ago by mkoeppe

  • Cc jipilab added
  • Status changed from needs_info to needs_review

Perhaps good enough for merging?

Note: See TracTickets for help on using tickets.