R installation fails on macOS with libcurl from Anaconda

[mkoeppe]

As reported in ​https://groups.google.com/d/msg/sage-devel/MMjoi2iDqo8/s55Uj2hHAwAJ (and earlier, in 2017, in ​https://groups.google.com/d/msg/sage-devel/aOK-ZtyH_Z8/Br4IycQnAgAJ).

Reproduced on macOS 10.14, Anaconda 3: config.log from R build:

configure:42569: checking for curl-config
configure:42587: found /anaconda3/bin/curl-config
configure:42599: result: /anaconda3/bin/curl-config
configure:42625: checking curl/curl.h usability
configure:42625: clang -c  -g -O2  -fPIC -I/anaconda3/include -I/usr/local/Cellar/pcre2/10.32/include   conftest.c >&5
configure:42625: $? = 0
configure:42625: result: yes
configure:42625: checking curl/curl.h presence
configure:42625: clang -E -I/anaconda3/include -I/usr/local/Cellar/pcre2/10.32/include   conftest.c
configure:42625: $? = 0
configure:42625: result: yes
configure:42625: checking for curl/curl.h
configure:42625: result: yes
configure:42639: checking if libcurl is version 7 and >= 7.22.0
configure:42668: clang -o conftest  -g -O2  -fPIC -I/anaconda3/include -I/usr/local/Cellar/pcre2/10.32/include   -L/Users/mkoeppe/s/sage/sage-rebasing/worktree-anaconda/local
/lib -Wl,-rpath,/Users/mkoeppe/s/sage/sage-rebasing/worktree-anaconda/local/lib  conftest.c -L/anaconda3/lib -lcurl -lssh2 -lssh2 -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 
-lresolv -lz -L/usr/local/Cellar/pcre2/10.32/lib -lpcre2-8 -lpcre -llzma -lbz2 -lz -licucore -ldl -lm  -liconv >&5
configure:42668: $? = 0
configure:42668: ./conftest
dyld: Library not loaded: @rpath/libcurl.4.dylib
  Referenced from: /Users/mkoeppe/s/sage/sage-rebasing/worktree-anaconda/local/var/tmp/sage/build/r-3.6.0/src/./conftest
  Reason: image not found
./configure: line 2254: 38949 Abort trap: 6           ./conftest$ac_exeext
configure:42668: $? = 134
configure: program exited with status 134

[mkoeppe]

All Anaconda libraries on macOS are run-path dependent:

$ otool -L /anaconda3/lib/libc*.*.dylib 
/anaconda3/lib/libcrypto.1.1.dylib:
	@rpath/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
/anaconda3/lib/libcurl.4.dylib:
	@rpath/libcurl.4.dylib (compatibility version 10.0.0, current version 10.0.0)
	@rpath/libssh2.1.dylib (compatibility version 1.0.0, current version 1.0.1)
	@rpath/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	@rpath/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
	@rpath/libgssapi_krb5.2.2.dylib (compatibility version 2.0.0, current version 2.2.0)
	@rpath/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)

[mkoeppe]

Perhaps build/pkgs/curl/spkg-configure.m4 (introduced in #24081) should check whether libcurl is actually usable?

[embray]

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

[dimpase]

Why do we see -I/usr/local/Cellar/pcre2/10.32/include ? Is it an attempt to mix Anaconda with Homebrew? That's sure to induce a severe hangover IMHO :-)

[embray]

I seem to recall mentioning on the mailing list (and a million other rants about OSX ;) that a frequent source of problems is mixing anaconda with homebrew with god knows what else.

Not even sure if that's really the problem here but it would be good to try to isolate these different packaging systems from each other to be sure.

[mkoeppe]

Replying to dimpase:

Why do we see -I/usr/local/Cellar/pcre2/10.32/include ? Is it an attempt to mix Anaconda with Homebrew? That's sure to induce a severe hangover IMHO :-)

Like every mac, it has bits and pieces from every pseudodistribution.

[mkoeppe]

I have pointed out a specific technical problem with rpath. The other mess is unrelated.

[mkoeppe]

Replying to mkoeppe:

I have pointed out a specific technical problem with rpath.

Which is, by the way, a frequent problem known to the net. See for example: ​http://cmake.3232098.n2.nabble.com/Issues-trying-to-use-the-Anaconda-compiler-tools-with-CMake-td7598056.html

[dimpase]

Isn't Anaconda coming with its own compiler, so that's iffy that everything gets to working correctly together?

Besides, I don't even know what that @rpath is meant to be.

[mkoeppe]

Replying to dimpase:

Isn't Anaconda coming with its own compiler, so that's iffy that everything gets to working correctly together?

The one that I installed didn't come with its own compiler; but the "conda build" magic is said to take care of the rpath stuff. Haven't tried.

Besides, I don't even know what that @rpath is meant to be.

​https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html

[mkoeppe]

Replying to embray:

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

Yes, let's do that.

[jhpalmieri]

Replying to embray:

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

The hope (​https://trac.sagemath.org/ticket/24081#comment:33) was that curl-config would be good enough, but it now makes sense to see if AC_CHECK_LIB helps.

[dimpase]

the problem with using foo-config, pkg-config --modversion foo, etc in spkg-configure.m4 files is that this won't work out of the box as soon as the library foo is not installed in a place known to the compliler/linker.

the conservative way would be indeed to resort to AC_CHECK_LIB etc, whereas the way to unvendor more would be to actually use the info one can extract from foo-config etc to supply the needed extra flags to the build process.

With PKG_CHECK_MODULES([FOO], [libfoo]...) it is in fact easy, as it automatically populates the custom-named variables with compiler and linker flags, FOO_CFLAGS and FOO_LIBS. So they can be written into sage-env-config and used by the dependencies of foo. Perhaps we could experiment with this approach with Homebrew-provided libraries that get installed into Cellar.

Anyhow, for curl one ought to use their ready autoconf macro, rather than reinvent the wheel: ​LIBCURL_CHECK_CONFIG

[mkoeppe]

Replying to dimpase:

With PKG_CHECK_MODULES([FOO], [libfoo]...) it is in fact easy, as it automatically populates the custom-named variables with compiler and linker flags, FOO_CFLAGS and FOO_LIBS.

Anaconda does provide pkg-config files, but unfortunately they do not include the rpath information either. Example, /anaconda3/lib/pkgconfig/libcurl.pc:

prefix=/anaconda3
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include
supported_protocols="DICT FILE FTP FTPS GOPHER HTTP HTTPS IMAP IMAPS POP3 POP3S RTSP SCP SFTP SMB SMBS SMTP SMTPS TELNET TFTP"
supported_features="SSL IPv6 UnixSockets libz AsynchDNS GSS-API SPNEGO Kerberos NTLM NTLM_WB TLS-SRP HTTPS-proxy"

Name: libcurl
URL: https://curl.haxx.se/
Description: Library to transfer files with ftp, http, etc.
Version: 7.64.0
Libs: -L${libdir} -lcurl
Libs.private: -lssh2 -lssh2 -lssl -lcrypto -lssl -lcrypto -lgssapi_krb5 -lresolv -lz
Cflags: -I${includedir} 

[mkoeppe]

Replying to jhpalmieri:

Replying to embray:

Perhaps so--it assumes that if curl-config works then so should the library. But maybe not? It should probably do an AC_CHECK_LIB too just to be safe.

The hope (​https://trac.sagemath.org/ticket/24081#comment:33) was that curl-config would be good enough, but it now makes sense to see if AC_CHECK_LIB helps.

Unfortunately the problem with rpaths can only be detected by linking and *runnning* a test program. AC_CHECK_LIB only links it, which will still succeed. Same with Dima's suggestion to use the macro from libcurl.m4 - it detects a "usable" libcurl, but if you link a program against it, it will still fail at runtime.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1af7d3c

Check whether libcurl links to executables

[mkoeppe]

With this patch it detects that libcurl from Anaconda actually doesn't work and then proceeds to build correctly. Runs and passes all tests.

[dimpase]

I'd like to understand how Anaconda actually is meant to be used. How on Earth an application using its libraries is meant to discover the info about @rpath ? Is one meant to use their special building tools?

In this sense Homebrew looks a much saner solution.

[mkoeppe]

Replying to dimpase:

I'd like to understand how Anaconda actually is meant to be used. How on Earth an application using its libraries is meant to discover the info about @rpath ? Is one meant to use their special building tools?

The default installation only puts $CONDA_PREFIX/bin into PATH and makes no further changes to build variables such as CFLAGS. I think it's only an oversight that binaries such as curl-config become available that leak information about build-time directories. Building software seems to be intended to be done with their package-oriented build tools.

[dimpase]

Perhaps Volker knows better - I recall a discussion where he proposed that Sage should be possible to build using (ana)conda. Now it looks harder than with more traditional OSX package systems such as Homebrew...

[vbraun]

@rpath is an Apple magic rpath prefix to search a special list of paths, see e.g. ​https://wincent.com/wiki/%40executable_path%2C_%40load_path_and_%40rpath

Since its an Apple proprietary thing I'm not sure if a lot of thought was give to it on the conda or curl side, its possibly just something that xcode coughs up.

It seems that specifying -rpath /path/to/conda/lib when linking to it would suffice, but then we'd need to know first whether we want to link to conda or not. As always, Apple is geared towards shipping binaries to users and not to compile stuff yourself.

[mkoeppe]

Need review.

[jhpalmieri]

This seems to work, in the sense that if I have anaconda installed with their stupid modification to my .profile, changing my PATH, then Sage thinks that curl is broken and so builds its own.

Maybe a better solution would be to use curl-config --prefix to actually locate the anaconda curl libraries and then use those if present. That's not happening with this branch, right? It would come closer to the goal of using whatever libraries are already installed. (Or alternatively, if anaconda is going to screw around with my PATH, they should also set some other environment variables telling programs where to find their headers and libraries.)

[mkoeppe]

Replying to jhpalmieri:

Maybe a better solution would be to use curl-config --prefix to actually locate the anaconda curl libraries and then use those if present. That's not happening with this branch, right? It would come closer to the goal of using whatever libraries are already installed. (Or alternatively, if anaconda is going to screw around with my PATH, they should also set some other environment variables telling programs where to find their headers and libraries.)

Yes, that's out of the scope of this ticket; it would go to #27699. It would likely not be curl-specific but rather set LDFLAGS to include an rpath directive.

[jhpalmieri]

I'm willing to give this a positive review, but I'm not sure I see the point. At least with my setup, once I allow anaconda to modify the path, other things besides R break. For example, tachyon:

gcc -Os -ffast-math -DBsd  -DUSEPNG    -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2-anaconda3/local/include  -c ../src/pngfile.c -o ../compile/macosx/libtachyon/pngfile.o
../src/pngfile.c:33:10: fatal error: 'png.h' file not found
#include "png.h" /* the libpng library headers */
         ^~~~~~~
1 error generated.
make[6]: *** [../compile/macosx/libtachyon/pngfile.o] Error 1

So this looks like one small fix to a much larger problem.

[dimpase]

maybe different versions of conda have different behaviour here, but on an installation I have, once you run "conda deactivate", the only addition to PATH contains conda and nothing else.

(base)~$ conda deactivate
$ echo $PATH
/home/dimpase/miniconda2/condabin:/usr/local/bin:/usr/bin:...
$ ls /home/dimpase/miniconda2/condabin/
conda

While building Sage on "active" conda is not working, merely having a non-active installation should not break stuff. IMHO.

whereas

$ conda activate
(base)~$ echo $PATH
/home/dimpase/miniconda2/bin:/home/dimpase/miniconda2/condabin:/usr/local/bin:...
(base)~$ which curl-config 
/home/dimpase/miniconda2/bin/curl-config

prepends more things to PATH.

[mkoeppe]

Replying to jhpalmieri:

I'm willing to give this a positive review, but I'm not sure I see the point. At least with my setup, once I allow anaconda to modify the path, other things besides R break. For example, tachyon:

gcc -Os -ffast-math -DBsd  -DUSEPNG    -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.8.rc2-anaconda3/local/include  -c ../src/pngfile.c -o ../compile/macosx/libtachyon/pngfile.o
../src/pngfile.c:33:10: fatal error: 'png.h' file not found
#include "png.h" /* the libpng library headers */
         ^~~~~~~
1 error generated.
make[6]: *** [../compile/macosx/libtachyon/pngfile.o] Error 1

So this looks like one small fix to a much larger problem.

Interesting. On my system, Tachyon compiled without problem.

[mkoeppe]

Replying to dimpase:

maybe different versions of conda have different behaviour here, but on an installation I have, once you run "conda deactivate", the only addition to PATH contains conda and nothing else.

This ticket is about fixing things when conda is "activated". It is activated after a default installation of Anaconda 3. I haven't tried with miniconda.

[jhpalmieri]

I have an old (I think) conda installation; it only added one directory to my PATH folder. Today when I ran conda init bash, it made more significant changes, and that may have other effects.

With the old version, I was seeing all sorts of strange things. I had built my own version of gmp, for example, but without C++ support. Sage built fine, and givaro in particular had no problems. When I allowed that one conda directory to be in the path, then the givaro build failed, saying that gmp did not have C++ support and should be built instead with --enable-cxx. The strange thing was that the version of gmp in conda had C++ support, and so did the version in SAGE_ROOT/local, but something about how conda was working led to the complaining that the file gmpxx.h was missing from /usr/local/include.

After I did conda init bash today, which makes a more serious change in my environment, the Sage build failed during brial, saying that gd wasn't found. gd is not listed as a dependency for brial, and I've never seen this come up before. I did make gd and then make, and brial built fine after that. In general, conda is making my computer act very strangely.

[dimpase]

well, I don't think we need to support an old conda.

[embray]

We don't need to support an old conda, but it would be nice for all of Sage's configure stuff to work well in a conda environment as well. It's not necessary for building Sage in Conda, but it will help. In the last week working with Isuru we've found a number of issues with Sage's new and improved/improving configure script as it relates to Conda. It's a useful test case as it does expose minor bugs in building in contexts where the usual assumptions (like prefix=/usr or prefix=/usr/local) do not apply.

[mkoeppe]

Replying to jhpalmieri:

After I did conda init bash today, which makes a more serious change in my environment, the Sage build failed during brial, saying that gd wasn't found. gd is not listed as a dependency for brial, and I've never seen this come up before. I did make gd and then make, and brial built fine after that.

BRiaL only seems to look for gd if it can't use m4ri with libpng - see ​https://github.com/BRiAl/BRiAl/blob/master/configure.ac#L81 So this also comes from the same libpng breakage that you mentioned above.

[embray]

Wow. That's so random...

[dimpase]

conda init ... is an experimental feature. I don't really know what it does. Cf. conda -h.

Anyhow, in the acticated conda environment the right thing would be to make it possible to use any conda-installed library/executable, and this ticket seems to be just a small step in this direction. (E.g. I doubt that GMP from conda will work).

Does anaconda have conda deactivate command?

[jhpalmieri]

At this point I have updated my anaconda installation. conda init bash appends this to my .bash_profile:

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/palmieri/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/Users/palmieri/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/Users/palmieri/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/Users/palmieri/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

Then conda activate/conda deactivate just modify PATH.

[dimpase]

I still don't understand why one would need to run conda init ....

Thanks for confirming that conda deactivate also works for anaconda. We ought to menion conda deactivate in the installation manual.

[jhpalmieri]

Replying to dimpase:

I still don't understand why one would need to run conda init ....

On OS X at least, when you install anaconda, it asks whether you want it to modify your shell scripts. If you say no, then whenever you want, you can run conda init ... to do that modification.

[dimpase]

but why would you need this?

can you use conda activate/deactivate instead?

[jhpalmieri]

With no modifications to my PATH:

$ ./anaconda3/condabin/conda activate

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

...

The modifications to .bash_profile also set some other variables:

CONDA_EXE="/Users/palmieri/anaconda3/bin/conda"
CONDA_PYTHON_EXE="/Users/palmieri/anaconda3/bin/python"
CONDA_SHLVL="0"

and it defines a conda command that is required to use with conda activate.

[dimpase]

OK, so it looks as if Anaconda needs a modified set of instructions, thanks. Hopefully the “inner parts” are the same as of (mini)conda...

[jhpalmieri]

conda init is mentioned in step 7 ​here. I see a similar instruction on the Linux installation page.

[dimpase]

both miniconda and anaconda use a package manager called Conda. Otherwise it seems they are not quite identical at the user interface level.

[embray]

Not in Sage 8.8. Let's please to try keep tickets' milestones related to the release in which we actually intend to include them, and in particular the release in which they were actually included, especially when closing tickets.