Opened 3 years ago

Closed 3 years ago

Last modified 2 years ago

#27941 closed defect (fixed)

R installation fails on macOS with libcurl from Anaconda

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-8.9
Component: packages: standard Keywords: conda
Cc: dimpase, embray, slelievre, jhpalmieri, vbraun, isuruf Merged in:
Authors: Matthias Koeppe Reviewers: John Palmieri
Report Upstream: N/A Work issues:
Branch: 1af7d3c (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description

As reported in https://groups.google.com/d/msg/sage-devel/MMjoi2iDqo8/s55Uj2hHAwAJ (and earlier, in 2017, in https://groups.google.com/d/msg/sage-devel/aOK-ZtyH_Z8/Br4IycQnAgAJ).

Reproduced on macOS 10.14, Anaconda 3: config.log from R build:

configure:42569: checking for curl-config
configure:42587: found /anaconda3/bin/curl-config
configure:42599: result: /anaconda3/bin/curl-config
configure:42625: checking curl/curl.h usability
configure:42625: clang -c  -g -O2  -fPIC -I/anaconda3/include -I/usr/local/Cellar/pcre2/10.32/include   conftest.c >&5
configure:42625: $? = 0
configure:42625: result: yes
configure:42625: checking curl/curl.h presence
configure:42625: clang -E -I/anaconda3/include -I/usr/local/Cellar/pcre2/10.32/include   conftest.c
configure:42625: $? = 0
configure:42625: result: yes
configure:42625: checking for curl/curl.h
configure:42625: result: yes
configure:42639: checking if libcurl is version 7 and >= 7.22.0
configure:42668: clang -o conftest  -g -O2  -fPIC -I/anaconda3/include -I/usr/local/Cellar/pcre2/10.32/include   -L/Users/mkoeppe/s/sage/sage-rebasing/worktree-anaconda/local
/lib -Wl,-rpath,/Users/mkoeppe/s/sage/sage-rebasing/worktree-anaconda/local/lib  conftest.c -L/anaconda3/lib -lcurl -lssh2 -lssh2 -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 
-lresolv -lz -L/usr/local/Cellar/pcre2/10.32/lib -lpcre2-8 -lpcre -llzma -lbz2 -lz -licucore -ldl -lm  -liconv >&5
configure:42668: $? = 0
configure:42668: ./conftest
dyld: Library not loaded: @rpath/libcurl.4.dylib
  Referenced from: /Users/mkoeppe/s/sage/sage-rebasing/worktree-anaconda/local/var/tmp/sage/build/r-3.6.0/src/./conftest
  Reason: image not found
./configure: line 2254: 38949 Abort trap: 6           ./conftest$ac_exeext
configure:42668: $? = 134
configure: program exited with status 134

Change History (47)

comment:1 Changed 3 years ago by mkoeppe

All Anaconda libraries on macOS are run-path dependent:

$ otool -L /anaconda3/lib/libc*.*.dylib 
/anaconda3/lib/libcrypto.1.1.dylib:
	@rpath/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/anaconda3/lib/libcurl.4.dylib:
	@rpath/libcurl.4.dylib (compatibility version 10.0.0, current version 10.0.0)
	@rpath/libssh2.1.dylib (compatibility version 1.0.0, current version 1.0.1)
	@rpath/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	@rpath/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	@rpath/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)
	@rpath/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

comment:2 Changed 3 years ago by mkoeppe

  • Cc jhpalmieri added

Perhaps build/pkgs/curl/spkg-configure.m4 (introduced in #24081) should check whether libcurl is actually usable?

comment:3 follow-ups: Changed 3 years ago by embray

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

comment:4 follow-up: Changed 3 years ago by dimpase

Why do we see -I/usr/local/Cellar/pcre2/10.32/include ? Is it an attempt to mix Anaconda with Homebrew? That's sure to induce a severe hangover IMHO :-)

comment:5 Changed 3 years ago by embray

I seem to recall mentioning on the mailing list (and a million other rants about OSX ;) that a frequent source of problems is mixing anaconda with homebrew with god knows what else.

Not even sure if that's really the problem here but it would be good to try to isolate these different packaging systems from each other to be sure.

comment:6 in reply to: ↑ 4 Changed 3 years ago by mkoeppe

Replying to dimpase:

Why do we see -I/usr/local/Cellar/pcre2/10.32/include ? Is it an attempt to mix Anaconda with Homebrew? That's sure to induce a severe hangover IMHO :-)

Like every mac, it has bits and pieces from every pseudodistribution.

comment:7 follow-up: Changed 3 years ago by mkoeppe

I have pointed out a specific technical problem with rpath. The other mess is unrelated.

comment:8 in reply to: ↑ 7 Changed 3 years ago by mkoeppe

Replying to mkoeppe:

I have pointed out a specific technical problem with rpath.

Which is, by the way, a frequent problem known to the net. See for example: http://cmake.3232098.n2.nabble.com/Issues-trying-to-use-the-Anaconda-compiler-tools-with-CMake-td7598056.html

comment:9 follow-up: Changed 3 years ago by dimpase

Isn't Anaconda coming with its own compiler, so that's iffy that everything gets to working correctly together?

Besides, I don't even know what that @rpath is meant to be.

comment:10 in reply to: ↑ 9 Changed 3 years ago by mkoeppe

Replying to dimpase:

Isn't Anaconda coming with its own compiler, so that's iffy that everything gets to working correctly together?

The one that I installed didn't come with its own compiler; but the "conda build" magic is said to take care of the rpath stuff. Haven't tried.

Besides, I don't even know what that @rpath is meant to be.

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html

comment:11 in reply to: ↑ 3 Changed 3 years ago by mkoeppe

Replying to embray:

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

Yes, let's do that.

comment:12 in reply to: ↑ 3 ; follow-up: Changed 3 years ago by jhpalmieri

Replying to embray:

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

The hope (https://trac.sagemath.org/ticket/24081#comment:33) was that curl-config would be good enough, but it now makes sense to see if AC_CHECK_LIB helps.

comment:13 follow-up: Changed 3 years ago by dimpase

the problem with using foo-config, pkg-config --modversion foo, etc in spkg-configure.m4 files is that this won't work out of the box as soon as the library foo is not installed in a place known to the compliler/linker.

the conservative way would be indeed to resort to AC_CHECK_LIB etc, whereas the way to unvendor more would be to actually use the info one can extract from foo-config etc to supply the needed extra flags to the build process.

With PKG_CHECK_MODULES([FOO], [libfoo]...) it is in fact easy, as it automatically populates the custom-named variables with compiler and linker flags, FOO_CFLAGS and FOO_LIBS. So they can be written into sage-env-config and used by the dependencies of foo. Perhaps we could experiment with this approach with Homebrew-provided libraries that get installed into Cellar.

Anyhow, for curl one ought to use their ready autoconf macro, rather than reinvent the wheel: LIBCURL_CHECK_CONFIG

comment:14 in reply to: ↑ 13 Changed 3 years ago by mkoeppe

Replying to dimpase:

With PKG_CHECK_MODULES([FOO], [libfoo]...) it is in fact easy, as it automatically populates the custom-named variables with compiler and linker flags, FOO_CFLAGS and FOO_LIBS.

Anaconda does provide pkg-config files, but unfortunately they do not include the rpath information either. Example, /anaconda3/lib/pkgconfig/libcurl.pc:

prefix=/anaconda3
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
supported_protocols="DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SCP SFTP SMB SMBS SMTP SMTPS TELNET TFTP"
supported_features="SSL IPv6 UnixSockets libz AsynchDNS GSS-API SPNEGO Kerberos NTLM NTLM_WB TLS-SRP HTTPS-proxy"

Name: libcurl
URL: https://curl.haxx.se/
Description: Library to transfer files with ftp, http, etc.
Version: 7.64.0
Libs: -L${libdir} -lcurl
Libs.private: -lssh2 -lssh2 -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 -lresolv -lz
Cflags: -I${includedir} 

comment:15 in reply to: ↑ 12 Changed 3 years ago by mkoeppe

Replying to jhpalmieri:

Replying to embray:

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

The hope (https://trac.sagemath.org/ticket/24081#comment:33) was that curl-config would be good enough, but it now makes sense to see if AC_CHECK_LIB helps.

Unfortunately the problem with rpaths can only be detected by linking and *runnning* a test program. AC_CHECK_LIB only links it, which will still succeed. Same with Dima's suggestion to use the macro from libcurl.m4 - it detects a "usable" libcurl, but if you link a program against it, it will still fail at runtime.

comment:16 Changed 3 years ago by mkoeppe

  • Branch set to u/mkoeppe/r_installation_fails_on_macos_with_libcurl_from_anaconda

comment:17 Changed 3 years ago by git

  • Commit set to 1af7d3cbc540e2e59959ba2ddf62c710a92b2d58

Branch pushed to git repo; I updated commit sha1. New commits:

1af7d3cCheck whether libcurl links to executables

comment:18 Changed 3 years ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Status changed from new to needs_review

With this patch it detects that libcurl from Anaconda actually doesn't work and then proceeds to build correctly. Runs and passes all tests.

comment:19 follow-up: Changed 3 years ago by dimpase

I'd like to understand how Anaconda actually is meant to be used. How on Earth an application using its libraries is meant to discover the info about @rpath ? Is one meant to use their special building tools?

In this sense Homebrew looks a much saner solution.

comment:20 in reply to: ↑ 19 Changed 3 years ago by mkoeppe

Replying to dimpase:

I'd like to understand how Anaconda actually is meant to be used. How on Earth an application using its libraries is meant to discover the info about @rpath ? Is one meant to use their special building tools?

The default installation only puts $CONDA_PREFIX/bin into PATH and makes no further changes to build variables such as CFLAGS. I think it's only an oversight that binaries such as curl-config become available that leak information about build-time directories. Building software seems to be intended to be done with their package-oriented build tools.

comment:21 Changed 3 years ago by dimpase

  • Cc vbraun added

Perhaps Volker knows better - I recall a discussion where he proposed that Sage should be possible to build using (ana)conda. Now it looks harder than with more traditional OSX package systems such as Homebrew...

comment:22 Changed 3 years ago by vbraun

@rpath is an Apple magic rpath prefix to search a special list of paths, see e.g. https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath

Since its an Apple proprietary thing I'm not sure if a lot of thought was give to it on the conda or curl side, its possibly just something that xcode coughs up.

It seems that specifying -rpath /path/to/conda/lib when linking to it would suffice, but then we'd need to know first whether we want to link to conda or not. As always, Apple is geared towards shipping binaries to users and not to compile stuff yourself.

comment:23 Changed 3 years ago by mkoeppe

Need review.

comment:24 Changed 3 years ago by dimpase

  • Cc isuruf added

comment:25 follow-up: Changed 3 years ago by jhpalmieri

This seems to work, in the sense that if I have anaconda installed with their stupid modification to my .profile, changing my PATH, then Sage thinks that curl is broken and so builds its own.

Maybe a better solution would be to use curl-config --prefix to actually locate the anaconda curl libraries and then use those if present. That's not happening with this branch, right? It would come closer to the goal of using whatever libraries are already installed. (Or alternatively, if anaconda is going to screw around with my PATH, they should also set some other environment variables telling programs where to find their headers and libraries.)

comment:26 in reply to: ↑ 25 Changed 3 years ago by mkoeppe

Replying to jhpalmieri:

Maybe a better solution would be to use curl-config --prefix to actually locate the anaconda curl libraries and then use those if present. That's not happening with this branch, right? It would come closer to the goal of using whatever libraries are already installed. (Or alternatively, if anaconda is going to screw around with my PATH, they should also set some other environment variables telling programs where to find their headers and libraries.)

Yes, that's out of the scope of this ticket; it would go to #27699. It would likely not be curl-specific but rather set LDFLAGS to include an rpath directive.

comment:27 follow-up: Changed 3 years ago by jhpalmieri

  • Reviewers set to John Palmieri
  • Status changed from needs_review to positive_review

I'm willing to give this a positive review, but I'm not sure I see the point. At least with my setup, once I allow anaconda to modify the path, other things besides R break. For example, tachyon:

gcc -Os -ffast-math -DBsd  -DUSEPNG    -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2-anaconda3/local/include  -c ../src/pngfile.c -o ../compile/macosx/libtachyon/pngfile.o
../src/pngfile.c:33:10: fatal error: 'png.h' file not found
#include "png.h" /* the libpng library headers */
         ^~~~~~~
1 error generated.
make[6]: *** [../compile/macosx/libtachyon/pngfile.o] Error 1

So this looks like one small fix to a much larger problem.

comment:28 follow-up: Changed 3 years ago by dimpase

maybe different versions of conda have different behaviour here, but on an installation I have, once you run "conda deactivate", the only addition to PATH contains conda and nothing else.

(base)~$ conda deactivate
$ echo $PATH
/home/dimpase/miniconda2/condabin:/usr/local/bin:/usr/bin:...
$ ls /home/dimpase/miniconda2/condabin/
conda

While building Sage on "active" conda is not working, merely having a non-active installation should not break stuff. IMHO.

whereas

$ conda activate
(base)~$ echo $PATH
/home/dimpase/miniconda2/bin:/home/dimpase/miniconda2/condabin:/usr/local/bin:...
(base)~$ which curl-config 
/home/dimpase/miniconda2/bin/curl-config

prepends more things to PATH.

Last edited 3 years ago by dimpase (previous) (diff)

comment:29 in reply to: ↑ 27 Changed 3 years ago by mkoeppe

Replying to jhpalmieri:

I'm willing to give this a positive review, but I'm not sure I see the point. At least with my setup, once I allow anaconda to modify the path, other things besides R break. For example, tachyon:

gcc -Os -ffast-math -DBsd  -DUSEPNG    -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2-anaconda3/local/include  -c ../src/pngfile.c -o ../compile/macosx/libtachyon/pngfile.o
../src/pngfile.c:33:10: fatal error: 'png.h' file not found
#include "png.h" /* the libpng library headers */
         ^~~~~~~
1 error generated.
make[6]: *** [../compile/macosx/libtachyon/pngfile.o] Error 1

So this looks like one small fix to a much larger problem.

Interesting. On my system, Tachyon compiled without problem.

comment:30 in reply to: ↑ 28 Changed 3 years ago by mkoeppe

Replying to dimpase:

maybe different versions of conda have different behaviour here, but on an installation I have, once you run "conda deactivate", the only addition to PATH contains conda and nothing else.

This ticket is about fixing things when conda is "activated". It is activated after a default installation of Anaconda 3. I haven't tried with miniconda.

comment:31 follow-up: Changed 3 years ago by jhpalmieri

I have an old (I think) conda installation; it only added one directory to my PATH folder. Today when I ran conda init bash, it made more significant changes, and that may have other effects.

With the old version, I was seeing all sorts of strange things. I had built my own version of gmp, for example, but without C++ support. Sage built fine, and givaro in particular had no problems. When I allowed that one conda directory to be in the path, then the givaro build failed, saying that gmp did not have C++ support and should be built instead with --enable-cxx. The strange thing was that the version of gmp in conda had C++ support, and so did the version in SAGE_ROOT/local, but something about how conda was working led to the complaining that the file gmpxx.h was missing from /usr/local/include.

After I did conda init bash today, which makes a more serious change in my environment, the Sage build failed during brial, saying that gd wasn't found. gd is not listed as a dependency for brial, and I've never seen this come up before. I did make gd and then make, and brial built fine after that. In general, conda is making my computer act very strangely.

comment:32 Changed 3 years ago by dimpase

well, I don't think we need to support an old conda.

comment:33 Changed 3 years ago by embray

We don't need to support an old conda, but it would be nice for all of Sage's configure stuff to work well in a conda environment as well. It's not necessary for building Sage in Conda, but it will help. In the last week working with Isuru we've found a number of issues with Sage's new and improved/improving configure script as it relates to Conda. It's a useful test case as it does expose minor bugs in building in contexts where the usual assumptions (like prefix=/usr or prefix=/usr/local) do not apply.

comment:34 in reply to: ↑ 31 Changed 3 years ago by mkoeppe

Replying to jhpalmieri:

After I did conda init bash today, which makes a more serious change in my environment, the Sage build failed during brial, saying that gd wasn't found. gd is not listed as a dependency for brial, and I've never seen this come up before. I did make gd and then make, and brial built fine after that.

BRiaL only seems to look for gd if it can't use m4ri with libpng - see https://github.com/BRiAl/BRiAl/blob/master/configure.ac#L81 So this also comes from the same libpng breakage that you mentioned above.

comment:35 Changed 3 years ago by embray

Wow. That's so random...

comment:36 Changed 3 years ago by dimpase

conda init ... is an experimental feature. I don't really know what it does. Cf. conda -h.

Anyhow, in the acticated conda environment the right thing would be to make it possible to use any conda-installed library/executable, and this ticket seems to be just a small step in this direction. (E.g. I doubt that GMP from conda will work).

Does anaconda have conda deactivate command?

comment:37 Changed 3 years ago by jhpalmieri

At this point I have updated my anaconda installation. conda init bash appends this to my .bash_profile:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/palmieri/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/palmieri/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/palmieri/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/palmieri/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Then conda activate/conda deactivate just modify PATH.

comment:38 follow-up: Changed 3 years ago by dimpase

I still don't understand why one would need to run conda init ....

Thanks for confirming that conda deactivate also works for anaconda. We ought to menion conda deactivate in the installation manual.

comment:39 in reply to: ↑ 38 Changed 3 years ago by jhpalmieri

Replying to dimpase:

I still don't understand why one would need to run conda init ....

On OS X at least, when you install anaconda, it asks whether you want it to modify your shell scripts. If you say no, then whenever you want, you can run conda init ... to do that modification.

comment:40 Changed 3 years ago by dimpase

but why would you need this?

can you use conda activate/deactivate instead?

comment:41 Changed 3 years ago by jhpalmieri

With no modifications to my PATH:

$ ./anaconda3/condabin/conda activate

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

...

The modifications to .bash_profile also set some other variables:

CONDA_EXE="/Users/palmieri/anaconda3/bin/conda"
CONDA_PYTHON_EXE="/Users/palmieri/anaconda3/bin/python"
CONDA_SHLVL="0"

and it defines a conda command that is required to use with conda activate.

comment:42 Changed 3 years ago by dimpase

OK, so it looks as if Anaconda needs a modified set of instructions, thanks. Hopefully the “inner parts” are the same as of (mini)conda...

comment:43 Changed 3 years ago by jhpalmieri

conda init is mentioned in step 7 here. I see a similar instruction on the Linux installation page.

comment:44 Changed 3 years ago by dimpase

both miniconda and anaconda use a package manager called Conda. Otherwise it seems they are not quite identical at the user interface level.

comment:45 Changed 3 years ago by vbraun

  • Branch changed from u/mkoeppe/r_installation_fails_on_macos_with_libcurl_from_anaconda to 1af7d3cbc540e2e59959ba2ddf62c710a92b2d58
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:46 Changed 3 years ago by embray

  • Milestone changed from sage-8.8 to sage-8.9

Not in Sage 8.8. Let's please to try keep tickets' milestones related to the release in which we actually intend to include them, and in particular the release in which they were actually included, especially when closing tickets.

comment:47 Changed 2 years ago by mkoeppe

  • Commit 1af7d3cbc540e2e59959ba2ddf62c710a92b2d58 deleted
  • Keywords conda added
Note: See TracTickets for help on using tickets.