Opened 14 months ago

Closed 4 weeks ago

#27699 closed enhancement (fixed)

configure: Exit with error if an Anaconda environment without compilers is active

Reported by: slelievre Owned by:
Priority: critical Milestone: sage-9.1
Component: build Keywords: conda
Cc: dimpase, saraedum, isuruf, jhpalmieri, embray, tscrim, fbissey Merged in:
Authors: Matthias Koeppe Reviewers: Isuru Fernando
Report Upstream: N/A Work issues:
Branch: 4ac6aaf (Commits) Commit: 4ac6aaf4cc0f91300ee31d31401631a6f4fc60e6
Dependencies: Stopgaps:

Description (last modified by mkoeppe)

This ticket makes it an error at configure time if an Anaconda environment is active that does not have the c-compiler, cxx-compiler, and fortran-compiler packages (or equivalent).

Many support requests come from the failure of Sage to build or to run in the presence of Anaconda.

Attachments (1)

Dockerfile-build-within-conda (1.7 KB) - added by mkoeppe 5 months ago.

Download all attachments as: .zip

Change History (60)

comment:1 follow-up: Changed 13 months ago by embray

I think it would be best to just make it possible to build sage in an anaconda env (at least given the right dependencies).

comment:2 in reply to: ↑ 1 Changed 12 months ago by mkoeppe

Replying to embray:

I think it would be best to just make it possible to build sage in an anaconda env (at least given the right dependencies).

+1

comment:3 Changed 12 months ago by mkoeppe

First ticket: #27941

comment:4 Changed 12 months ago by embray

  • Milestone sage-8.8 deleted

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

comment:5 Changed 12 months ago by mkoeppe

As discussed in #27941, if Anaconda is detected, it may suffice to add an -rpath directive to our LDFLAGS to enable linking against the libraries provided by Anaconda.

comment:6 Changed 7 months ago by mkoeppe

It suffices to install the following conda packages:

  • c-compiler
  • cxx-compiler
  • fortran-compiler

They take care of adding the necessary flags. See https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html (end) and also #28745.

comment:7 Changed 5 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Description modified (diff)
  • Milestone set to sage-9.1
  • Type changed from task to enhancement

comment:8 Changed 5 months ago by mkoeppe

  • Branch set to public/27699-detect-anaconda

comment:9 Changed 5 months ago by mkoeppe

  • Cc dimpase saraedum isuruf added
  • Commit set to f75a807db54b3d602332a5927c73317bfd6bffd6

New commits:

f75a807configure.ac, m4/sage_check_conda_compilers.m4: Check for conda

comment:10 Changed 5 months ago by mkoeppe

  • Status changed from new to needs_review

comment:11 Changed 5 months ago by mkoeppe

  • Cc jhpalmieri added

comment:12 follow-up: Changed 5 months ago by dimpase

Nice, but does it always do the job? E.g. I noticed that I must comment out everything in the following block inserted by conda into my .bashrc:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/scratch2/dimpase/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
 if [ $? -eq 0 ]; then
    eval "$__conda_setup"
 else
    if [ -f "/home/scratch2/dimpase/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/home/scratch2/dimpase/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/scratch2/dimpase/miniconda3/bin:$PATH"
    fi
fi
  • deactivating conda is not enough to let a conda-less build of Sage complete on Fedora.
Last edited 5 months ago by dimpase (previous) (diff)

comment:13 in reply to: ↑ 12 Changed 5 months ago by mkoeppe

Replying to dimpase:

Nice, but does it always do the job? E.g. I noticed that I must comment out everything in the following block inserted by conda into my .bashrc:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !! ...
  • deactivating conda is not enough to let a conda-less build of Sage complete on Fedora.

That would be a conda bug. What is the environment on that system after conda deactivate?

comment:14 Changed 5 months ago by dimpase

If I leave these things uncommented, and ssh into the box, I land in conda's (base). Then

(base) clpc171[~]$ conda deactivate  
clpc171[~]$ 
clpc171[~]$ logout
Connection to clpc171.cs.ox.ac.uk closed.
coml0531@raven:~$ ssh cs
Last login: Mon Jan 13 14:07:02 2020 from raven.linux.ox.ac.uk
(base) clpc171[~]$ 

and I land back into conda, as you see...

comment:15 Changed 5 months ago by mkoeppe

That's normal. conda deactivate and conda activate manipulate the environment of the current shell.

comment:16 Changed 5 months ago by dimpase

but then every sh call in a build that happens to source ~/.bashrc activates conda, resulting in broken builds.

comment:17 Changed 5 months ago by dimpase

E.g.:

clpc171[~]$ bash
(base) clpc171[~]$ 

Given that bash is called in a lot of places in build/ we obviously have a problem.

comment:18 follow-up: Changed 5 months ago by mkoeppe

The conda initialization code belongs into ~/.bash_profile, not ~/.bashrc. This is where conda init puts it. It is supposed to be invoked for login shells only.

comment:19 Changed 5 months ago by dimpase

I have not done anything special to install conda on that box. Perhaps it just has no .bash_profile

I will check as soon as I am back to keyboard. Feels like a conda bug to me.

comment:20 Changed 5 months ago by mkoeppe

It may be behavior of older versions of conda. In any case, also ~/.bashrc is only run by interactive bash invocations. If one of our build scripts calls it, that's a bug on our side.

comment:21 in reply to: ↑ 18 Changed 5 months ago by mkoeppe

Replying to mkoeppe:

The conda initialization code belongs into ~/.bash_profile, not ~/.bashrc. This is where conda init puts it. It is supposed to be invoked for login shells only.

I think it may be a difference in conda init behavior between mac and linux.

comment:22 follow-up: Changed 5 months ago by dimpase

I do have ~/.bash_profile - but it was not used by conda init (which I did last month, so it's a pretty recent install).

I'll move that chunk to ~/.bash_profile and verify whether it allows to build.

Isuru, is this not a conda bug, installing that stuff into ~/.bashrc ?

comment:23 in reply to: ↑ 22 Changed 5 months ago by isuruf

Isuru, is this not a conda bug, installing that stuff into ~/.bashrc ?

You could do,

conda config --set auto_activate_base false

to make it not initialize by default.

comment:25 follow-ups: Changed 5 months ago by dimpase

ok, so this stuff has to be in Sage's docs.

one way or another, why would anyone want automatic activation of conda in non-login shell?

comment:26 Changed 5 months ago by jhpalmieri

For what it's worth, I very recently installed anaconda on a Mac, and the installation wrote to .bash_profile, not .bashrc.

comment:27 follow-up: Changed 5 months ago by mkoeppe

FWIW, I successfully built sage from scratch after conda deactivate. But this is with a miniconda installation with a clean base environment.

I'll test again on Linux with a dirty base environment that has lots of bin/*-config executables - this was the problem in #27941.

comment:28 in reply to: ↑ 25 ; follow-up: Changed 5 months ago by mkoeppe

Replying to dimpase:

one way or another, why would anyone want automatic activation of conda in non-login shell?

I don't know but it does not matter as long as it's not also done in noninteractive shells (which it isn't).

comment:29 Changed 5 months ago by mkoeppe

  • Keywords conda added

comment:30 Changed 5 months ago by mkoeppe

  • Keywords Anaconda removed

comment:31 in reply to: ↑ 25 Changed 5 months ago by mkoeppe

Replying to dimpase:

ok, so this stuff has to be in Sage's docs.

I'll update the doc in this ticket.

comment:32 Changed 5 months ago by git

  • Commit changed from f75a807db54b3d602332a5927c73317bfd6bffd6 to d5f5e9aa2cf58debe727d3867719b1ab65a575a8

Branch pushed to git repo; I updated commit sha1. New commits:

d5f5e9asrc/doc/en/installation/source.rst: Add notes on using Anaconda/Miniconda

Changed 5 months ago by mkoeppe

comment:33 in reply to: ↑ 27 ; follow-up: Changed 5 months ago by mkoeppe

Replying to mkoeppe:

I'll test again on Linux with a dirty base environment that has lots of bin/*-config executables - this was the problem in #27941.

See the attached Dockerfile for this setup.

comment:34 in reply to: ↑ 28 ; follow-up: Changed 5 months ago by dimpase

Replying to mkoeppe:

Replying to dimpase:

one way or another, why would anyone want automatic activation of conda in non-login shell?

I don't know but it does not matter as long as it's not also done in noninteractive shells (which it isn't).

Well, I can imagine a Sage test invoking an interactive shell, and then, well, all the bets are off. Can we test that Conda has auto_activate_base=false - if we are outside conda?

As well, the docs ought to mention comment:23 (atm they only tell how to use conda, but they also ought to tell how not to)

comment:35 in reply to: ↑ 34 Changed 5 months ago by mkoeppe

Replying to dimpase:

Replying to mkoeppe:

Replying to dimpase:

one way or another, why would anyone want automatic activation of conda in non-login shell?

I don't know but it does not matter as long as it's not also done in noninteractive shells (which it isn't).

Well, I can imagine a Sage test invoking an interactive shell, and then, well, all the bets are off.

If there is such a bug in the testsuite, we'll fix it.

This really isn't specific to conda. Users' interactive shell configuration in ~/.bashrc could also define all kinds of incompatible aliases or print arbitrary output etc. that could confuse a doctest.

Can we test that Conda has auto_activate_base=false - if we are outside conda?

I don't know what you mean by that.

As well, the docs ought to mention comment:23 (atm they only tell how to use conda, but they also ought to tell how not to)

The added docs already mention conda deactivate. And that is sufficient.

Telling the user that they have to disable conda by editing their init scripts, or uninstall conda, or change their conda configuration --- all of this is going in the wrong direction for Sage.

comment:36 Changed 4 months ago by mkoeppe

  • Cc embray added

comment:37 in reply to: ↑ 33 Changed 4 months ago by mkoeppe

Replying to mkoeppe:

Replying to mkoeppe:

I'll test again on Linux with a dirty base environment that has lots of bin/*-config executables - this was the problem in #27941.

See the attached Dockerfile for this setup.

With #29026 (which fixes a compile error in the conda environment reported in #29005), this works fine.

Last edited 4 months ago by mkoeppe (previous) (diff)

comment:38 Changed 4 months ago by mkoeppe

I will not add #29026 as a dependency because the main purpose of the present ticket is to prevent broken builds in the situation described in the ticket.

Needs review.

comment:39 Changed 4 months ago by mkoeppe

  • Cc tscrim added

comment:40 Changed 4 months ago by mkoeppe

  • Cc fbissey added

comment:41 Changed 4 months ago by isuruf

  • Status changed from needs_review to needs_work

This would break building the conda package.

comment:42 Changed 4 months ago by mkoeppe

When building the conda package, are the c-compiler, cxx-compiler, and fortran-compiler packages not installed?

comment:43 follow-up: Changed 4 months ago by isuruf

Nope. Individual packages like gcc_linux-64 (or clang_osx-64) are installed.

comment:44 Changed 4 months ago by mkoeppe

Would you have a suggestion how we should test this more generally?

comment:45 in reply to: ↑ 43 ; follow-up: Changed 6 weeks ago by mkoeppe

Replying to isuruf:

Individual packages like gcc_linux-64 (or clang_osx-64) are installed.

In that case, are the environment variables CC and CXX guaranteed to be set?

comment:46 Changed 6 weeks ago by git

  • Commit changed from d5f5e9aa2cf58debe727d3867719b1ab65a575a8 to d566fdd8d24012d41872ec654c05b2d926f31c5c

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f854dd2configure.ac, m4/sage_check_conda_compilers.m4: Check for conda
d566fddsrc/doc/en/installation/source.rst: Add notes on using Anaconda/Miniconda

comment:47 Changed 6 weeks ago by git

  • Commit changed from d566fdd8d24012d41872ec654c05b2d926f31c5c to 193593a858ab42500d7095e17fe140a5810296f1

Branch pushed to git repo; I updated commit sha1. New commits:

193593am4/sage_check_conda_compilers.m4: Check environment variables CC, CXX, FC

comment:48 Changed 6 weeks ago by mkoeppe

  • Description modified (diff)
  • Status changed from needs_work to needs_review

comment:49 in reply to: ↑ 45 Changed 6 weeks ago by isuruf

Replying to mkoeppe:

Replying to isuruf:

Individual packages like gcc_linux-64 (or clang_osx-64) are installed.

In that case, are the environment variables CC and CXX guaranteed to be set?

Yes.

comment:50 Changed 5 weeks ago by mkoeppe

Then is the updated branch OK for building the conda package?

comment:51 Changed 5 weeks ago by mkoeppe

  • Priority changed from major to critical

comment:52 Changed 5 weeks ago by mkoeppe

  • Summary changed from Detect Anaconda to configure: Exit with error if an Anaconda environment without compilers is active

comment:53 Changed 5 weeks ago by mkoeppe

@isuruf Any objections?

comment:54 Changed 5 weeks ago by isuruf

  • Reviewers set to Isuru Fernando
  • Status changed from needs_review to positive_review

comment:55 Changed 5 weeks ago by mkoeppe

Thank you!

comment:56 Changed 5 weeks ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:57 Changed 5 weeks ago by git

  • Commit changed from 193593a858ab42500d7095e17fe140a5810296f1 to 4ac6aaf4cc0f91300ee31d31401631a6f4fc60e6

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

234d23cconfigure.ac, m4/sage_check_conda_compilers.m4: Check for conda
be7b446src/doc/en/installation/source.rst: Add notes on using Anaconda/Miniconda
4ac6aafm4/sage_check_conda_compilers.m4: Check environment variables CC, CXX, FC

comment:58 Changed 5 weeks ago by mkoeppe

  • Status changed from needs_work to positive_review

rebased on 9.1.rc2

comment:59 Changed 4 weeks ago by vbraun

  • Branch changed from public/27699-detect-anaconda to 4ac6aaf4cc0f91300ee31d31401631a6f4fc60e6
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.