Add SAGE_ROOT/environment.yml for "conda env create -f".

[mkoeppe]

These files describes the conda environment (conda package dependencies):

environment.yml - describes conda packages that are useful for installing sage.

This would be a more idiomatic alternative to the conda install line that is shown at the end of ./configure

In combination with the spkg-configure mechanism, this will speed up installation of Sage, in particular for users stuck on outdated OS distributions.

Open:

• use GitLab CI to actually check that sage builds with environment.yml
• scripts to support keeping package versions in sync between environment.yml, sage-the-distribution (build/pkgs), and conda-forge packaging (​https://github.com/conda-forge/sage-feedstock/tree/master/recipe, ​https://github.com/conda-forge/sagelib-feedstock/tree/master/recipe)

See also:

• #28752 for a more ambitious conda ticket
• #26668 Add documentation for implementing spkg-configure.m4 for SPKGs
• #27330 Try to use as many system packages as possible

[mkoeppe]

New commits:

​cf6e48a

First version of environment.yml for conda

[saraedum]

mkoeppe: I can only see one file in the diff here on trac. Did you forget to add the other?

[mkoeppe]

This is how far I got!

[mkoeppe]

Any help and discussion is welcome.

[saraedum]

First of all, I think it's great to explore ways to provide an alternative to the SPKG system. I have been wanting to use conda for this for a long time.

I have not used conda-debug a lot but I think a combination of conda debug [sagelib-recipe], conda install sage, conda remove sagelib --force should essentially do the right thing here as well.

A proper environment.yml file, as you propose, probably makes more sense though. For me the question is how we can make sure that it is kept up to date. We might want to use GitLab CI to actually check that sagelib builds with that environment.yml.

In the long run, it would be great if doctests passed, see ​https://github.com/conda-forge/sagelib-feedstock/issues/19.

[saraedum]

I am not sure I understand your intentions for the sage-the-distribution environment.yml. What would the workflow look like? Install that environment.yml, then force-uninstall the package that I want to work on? How would the SPKG system know which packages it needs to build now?

[mkoeppe]

Replying to saraedum:

I am not sure I understand your intentions for the sage-the-distribution environment.yml. What would the workflow look like? Install that environment.yml, then force-uninstall the package that I want to work on? How would the SPKG system know which packages it needs to build now?

Through the spkg-configure mechanism that was introduced recently, sage-the-distribution decides whether to install its own version of the package; but this can be overridden using configure options such as --with-system-glpk=force. (Here "system", of course, from the viewpoint of sage, includes the conda packages.) Sage developers could use the conda environment to avoid compiling many packages by default, but still work on and install sage-the-distribution packages that override the packages provided by conda.

[mkoeppe]

Replying to saraedum:

We might want to use GitLab CI to actually check that sagelib builds with that environment.yml.

+1

[saraedum]

So, to make this actually useful, we'd need the corresponding configure flags to use "system" for everything that is in environment.yml as well, right?

[saraedum]

But maybe we should split this into two tickets. One to do the sagelib environment.yml, and then this one to do the more complicated sage-the-distribution environment.yml.

[mkoeppe]

Replying to saraedum:

So, to make this actually useful, we'd need the corresponding configure flags to use "system" for everything that is in environment.yml as well, right?

No, the default is already to use the "system" package when it is available and deemed suitable by the configure test. It can be changed to using the sage distribution's package by using configure --with-system-glpk=no.

[mkoeppe]

Replying to saraedum:

But maybe we should split this into two tickets. One to do the sagelib environment.yml, and then this one to do the more complicated sage-the-distribution environment.yml.

Good idea. I have created #28752 for that and have narrowed down the present ticket.

[mkoeppe]

Replying to mkoeppe:

Replying to saraedum:

But maybe we should split this into two tickets. One to do the sagelib environment.yml, and then this one to do the more complicated sage-the-distribution environment.yml.

Good idea. I have created #28752 for that and have narrowed down the present ticket.

(Actually, the sagelib one is more complicated - see #28752!)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3e95cab

environment.yml: Add patch, ninja

[mkoeppe]

This works reasonably well already (this is on macOS Mojave, with current miniconda 3). I had the problem that automake and aclocal installed by conda segfault, so I bootstrapped Sage using the homebrew binaries. Then, running configure, most packages that have spkg-configure scripts recognize the installed conda packages, with the following exceptions:

SAGE_SPKG_CONFIGURE_PERL_TERM_READLINE_GNU
SAGE_SPKG_CONFIGURE_PARI_SEADATA_SMALL
SAGE_SPKG_CONFIGURE_GF2X
SAGE_SPKG_CONFIGURE_LIBSEMIGROUPS
SAGE_SPKG_CONFIGURE_PATCH
SAGE_SPKG_CONFIGURE_ISL
SAGE_SPKG_CONFIGURE_PARI*
SAGE_SPKG_CONFIGURE_ECLIB
SAGE_SPKG_CONFIGURE_LCALC

Then, a parallel make succesfully compiled 135 packages but got stuck on ecl-16.1.2.p5, openblas-0.3.6.p0, and tachyon-0.98.9.p7.

[isuruf]

For pari, you need to install pari-nftables as well. Can you post the config.log so that we can debug why the other packages are not picked up?

Btw, since this ticket focuses on spkg-configure, shouldn't the packages installed be limited to the ones with an spkg-configure?

[mkoeppe]

Replying to isuruf:

since this ticket focuses on spkg-configure, shouldn't the packages installed be limited to the ones with an spkg-configure?

Good point.

[saraedum]

mkoeppe, I think your environment.yml missed the pins from ​https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/master/recipe/conda_build_config.yaml. I am adding them now.

[saraedum]

mkoeppe, I update the environment yml. The only ones that don't work for me now are:

• perl_term_readline_gnu
• gp2c (from PARI)

isuruf: are they packaged somewhere on conda-forge?

Also I have to rely on the system's pkgconf. Do we have pkgconf in conda-forge?

---

New commits:

​00745e1	Merge remote-tracking branch 'origin/develop' into u/mkoeppe/worktree-sagelib-conda
​3c693d0	environment.yml that mostly passes ./configure

[saraedum]

I'm building sage now with this environment.yml. Let's see how far I get…

[mkoeppe]

Thanks a lot, this is looking great.

As for perl_term_readline_gnu, as far as I understand, conda-forge packages perl and perl-app-cpanminus but not individual CPAN packages, and one is expected to install any packages using cpanminus.

[isuruf]

perl_term_readline_gnu and gp2c are not packaged. pkgconf is in conda-forge as pkg-config

[isuruf]

If you are on Debian or Ubuntu, you need to fix ​https://trac.sagemath.org/ticket/28533

[saraedum]

Replying to isuruf: pkgconf is in conda-forge as pkg-config

Is this really the same thing? There's at least no pkgconf binary in pkg-config. And judging from my ArchLinux?'s pkgconf and conda-forge's pkg-config --help, these seem to be two different things.

[saraedum]

Replying to saraedum:

I'm building sage now with this environment.yml. Let's see how far I get…

pkgconf and tachyon failed to build.

[saraedum]

SageMath sets the following for tachyon:

+PNGINC= -I$(SAGE_LOCAL)/include
+PNGLIB= -L$(SAGE_LOCAL)/lib -lpng -lz

Reporting to #27186 where the m4 script for libpng was added.

[saraedum]

Tachyon's build is very confusing. It appears that they are ignoring the environment's CFLAGS.

[isuruf]

Replying to saraedum:

Replying to isuruf: pkgconf is in conda-forge as pkg-config

Is this really the same thing? There's at least no pkgconf binary in pkg-config. And judging from my ArchLinux?'s pkgconf and conda-forge's pkg-config --help, these seem to be two different things.

That's fine. They both do the same thing. ​https://github.com/pkgconf/pkgconf#compatibility-with-pkg-config

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6c2443

use conda-forge's pkg-config

[isuruf]

Replying to mkoeppe:

but not individual CPAN packages

It's just that they haven't been package yet. If there's anybody who is interested in helping out, they can be packaged.

[isuruf]

We can add openblas, cliquer, r-base to the list now.

[isuruf]

New commits:

​cfeb376

Add configure deps and more packages

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​256e1f5

Need blas-devel for cblas.so symlinks

[dimpase]

The latter doesn't work for me:

$ conda env create -n sage -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - blas-devel[build=*openblas]

[mkoeppe]

Replying to dimpase:

The latter doesn't work for me:

$ conda env create -n sage -f environment.yml
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound: 
  - blas-devel[build=*openblas]

Neither for me.

[mkoeppe]

After removing this line, I get the following:

[python3-3.7.3.p1] /opt/conda/envs/sage/bin/x86_64-conda_cos6-linux-gnu-cc -pthread -shared -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib -L. -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib -L. -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/envs/sage/lib -Wl,-rpath-link,/opt/conda/envs/sage/lib -L/opt/conda/envs/sage/lib -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/envs/sage/include build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_ctypes/_ctypes.o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_ctypes/callbacks.o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_ctypes/callproc.o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_ctypes/stgdict.o build/temp.linux-x86_64-3.7/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Modules/_ctypes/cfield.o -L. -L/sage/local/lib -L/opt/conda/envs/sage/lib -L/usr/local/lib -lffi -ldl -lpython3.7m -o build/lib.linux-x86_64-3.7/_ctypes.cpython-37m-x86_64-linux-gnu.so
[python3-3.7.3.p1] *** WARNING: renaming "_crypt" since importing it failed: build/lib.linux-x86_64-3.7/_crypt.cpython-37m-x86_64-linux-gnu.so: undefined symbol: crypt_r
...
[python3-3.7.3.p1] hashlib module imported OK
[python3-3.7.3.p1] Traceback (most recent call last):
[python3-3.7.3.p1]   File "<string>", line 1, in <module>
[python3-3.7.3.p1]   File "/sage/local/var/tmp/sage/build/python3-3.7.3.p1/src/Lib/crypt.py", line 3, in <module>
[python3-3.7.3.p1]     import _crypt
[python3-3.7.3.p1] ModuleNotFoundError: No module named '_crypt'
[python3-3.7.3.p1] crypt module failed to import
[python3-3.7.3.p1] readline module imported OK
[python3-3.7.3.p1] socket module imported OK
[python3-3.7.3.p1] Error: One or more modules failed to import.

This is on a Debian docker image based on continuumio/miniconda3:latest - see attached

[mkoeppe]

I have created #29012 for this defect.

[mkoeppe]

Would it be a good idea to generate this file from information to be put in build/pkgs/SPKG/conda-environment.txt as in #29053?

---

New commits:

​c01f4f8	Merge tag '9.1.beta0' into t/28745/worktree-sagelib-conda
​a511c83	add openssl, remove blas-devel

[mkoeppe]

If one works around the python build failures of #29012 by using system python3 (#27824), several other errors show up:

Testsuite of libatomic_ops:

  [libatomic_ops-7.6.2]   /bin/bash ../libtool  --tag=CC   --mode=link /opt/conda/bin/x86_64-conda_cos6-linux-gnu-cc  -Wall -Wextra -Wpedantic -Wno-long-long -O2 -g -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/conda/include   -L/sage/local/lib -Wl,-rpath,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -o test_malloc test_malloc.o -lpthread ../src/libatomic_ops_gpl.la ../src/libatomic_ops.la 
  [libatomic_ops-7.6.2]   libtool: link: /opt/conda/bin/x86_64-conda_cos6-linux-gnu-cc -Wall -Wextra -Wpedantic -Wno-long-long -O2 -g -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/conda/include -Wl,-rpath -Wl,/sage/local/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath -Wl,/opt/conda/lib -Wl,-rpath-link -Wl,/opt/conda/lib -o test_malloc test_malloc.o  -L/sage/local/lib -L/opt/conda/lib -lpthread ../src/.libs/libatomic_ops_gpl.a /sage/local/var/tmp/sage/build/libatomic_ops-7.6.2/src/src/.libs/libatomic_ops.a ../src/.libs/libatomic_ops.a
  [libatomic_ops-7.6.2]   /opt/conda/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: ../src/.libs/libatomic_ops_gpl.a(atomic_ops_stack.o): in function `AO_double_compare_and_swap_release':
  [libatomic_ops-7.6.2]   /sage/local/var/tmp/sage/build/libatomic_ops-7.6.2/src/src/atomic_ops/sysdeps/gcc/generic.h:219: undefined reference to `__atomic_compare_exchange_16'
  [libatomic_ops-7.6.2]   collect2: error: ld returned 1 exit status
  [libatomic_ops-7.6.2]   make[7]: *** [Makefile:634: test_malloc] Error 1

[mkoeppe]

... as well as giac and fflas_ffpack.

To reproduce:

EXTRA_DOCKER_BUILD_ARGS="--build-arg TARGETS=\"build ptest\" --build-arg USE_MAKEFLAGS=\"-k V=0\"" tox -e docker-conda-forge-standard

on the branch of #27824.

[mkoeppe]

(see also #29327)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​4decf97	First version of environment.yml for conda
​ca0cb5f	environment.yml: Add patch, ninja
​6b42c51	environment.yml that mostly passes ./configure
​d762508	use conda-forge's pkg-config
​13123dd	Add configure deps and more packages
​68219bb	Need blas-devel for cblas.so symlinks
​5192b93	add openssl, remove blas-devel
​ba3458b	build/pkgs/conda.txt: Use python<3.8

[mkoeppe]

Shall we try to get this one here to work for 9.1?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​12a00ba

Merge tag '9.1.rc2' into t/28745/worktree-sagelib-conda

[mkoeppe]

Moving some tickets to 9.2. This is not a promise that I will be working on them.

[isuruf]

I added SAGE_ROOT/environment.yml for spkg-configure ones and SAGE_ROOT/src/environment.yml for all standard packages. These files are auto generated. Not sure if we should check them into the repo for ease of use.

[mkoeppe]

I don't think the generated files should be checked in - instead they should be added to the list of files that are put into the configure tarball.

[mkoeppe]

See for example commit 885092e77b06b6d616297edd8ac1a37db9113cc6, where files were added

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​61b8fca

add environment files to source tarball

[mkoeppe]

Also, if you could untabify src/doc/bootstrap...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c252e72

replace tabs with 4 spaces

[vbraun]

Merge conflict

[mkoeppe]

New commits:

​025064e

Merge tag '9.3.beta0' into t/28745/environment_yaml

[isuruf]

New commits:

​7def5b9

Merge remote-tracking branch 'trac/develop' into u/mkoeppe/environment_yaml

[vbraun]

Merge conflict

[isuruf]

Which branch should I fix the conflict against? develop doesn't seem to have a conflict with this branch

[dimpase]

Replying to isuruf:

Which branch should I fix the conflict against? develop doesn't seem to have a conflict with this branch

it's unfortunate feauture of our workflow that this is a bit of a moving target. ​https://github.com/vbraun/sage/tree/develop

do not try to merge/rebase with it, as it's likely to be in vain.

[mkoeppe]

Last 10 new commits:

​e61929d	build/pkgs/sphinx/install-requires.txt: Update from gentoo
​8fe0e35	build/pkgs/python_openid: Unused, remove
​ba1d913	build/pkgs/itsdangerous: Unused, remove
​bb6c4ae	build/pkgs/tox/install-requires.txt: New
​d507501	build/pkgs/networkx/install-requires.txt: Update from gentoo
​c3a9352	build/pkgs/pathpy/install-requires.txt: Package removed in #30611, remove
​ebd4610	build/pkgs/pynac/install-requires.txt: New
​6ec00dd	build/pkgs/{numpy,pillow}/install-requires.txt: New
​77cc105	Merge commit '6ec00dd002b41c4fe9b91233f2cebe37c87144d9' of git://trac.sagemath.org/sage into t/28745/environment_yaml
​e53623d	build/pkgs/itsdangerous/distros/conda.txt: Remove

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. Last 10 new commits:

​d365707	build/pkgs/packaging: Update to 20.4
​248c84a	build/pkgs/pathpy: No longer required by pickleshare, remove
​be5bdf2	build/pkgs/pyparsing: Update to 2.4.7
​88b8427	build/pkgs/pytz: Update to 2020.4
​df7972c	build/pkgs/rpy2: Update to 3.3.6
​7cd96b2	build/pkgs/scipy/patches/extern_decls.patch: Remove
​84d778c	Fixing upstream url for networkx
​13af6c6	fix tarball name
​45b3408	Merge commit '13af6c6b8c1533ba9d1b45b127e1a7b7d30000c6' of git://trac.sagemath.org/sage into t/28745/environment_yaml
​736e006	build/pkgs/pathpy/distros/conda.txt: Remove