Opened 3 years ago

Closed 3 years ago

Last modified 3 years ago

#26899 closed defect (fixed)

OS X Mojave without /usr/include: a range of issues

Reported by: jhpalmieri Owned by:
Priority: blocker Milestone: sage-8.8
Component: packages: standard Keywords:
Cc: embray Merged in:
Authors: Dima Pasechnik, Volker Braun Reviewers: Dima Pasechnik, Volker Braun
Report Upstream: Reported upstream. No feedback yet. Work issues:
Branch: db50d09 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by dimpase)

On OS X Mojave, it is possible to have Xcode installed but to have no directory /usr/include.

  • The configure check from #26286 finds zlib somewhere, but the Python build does not: Python reports that it has built without zlib support. Without zlib, pip will not build.
  • gcc/gfortran needs the location of headers to be explicitly passed via --with-native-system-header-dir= option. (untested)

Attachments (3)

python2-2.7.15.p0.log (1.3 MB) - added by jhpalmieri 3 years ago.
sage.config.log (94.2 KB) - added by jhpalmieri 3 years ago.
python2.config.log (469.7 KB) - added by jhpalmieri 3 years ago.

Download all attachments as: .zip

Change History (96)

comment:1 Changed 3 years ago by dimpase

See #26713. I'd close this one, but yes, what's offered on #26713 is only a temporary solution. We need to think what to do with OSX (Homebrew? (Ana)conda?), it's getting too hard to jump through new and new hoops built up by Apple.

comment:2 Changed 3 years ago by jhpalmieri

On this machine, Sage 8.5.beta4 builds. I am now testing. So it's probably not the same issue as #26713 which was opened before 8.5.rc0 was released. Or it's not necessarily the same issue. In particular, I believe that the problem on this machine can be traced directly to #26286.

I believe that on this machine, I once ran

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg 

as advised at #26713, and then I believe a subsequent Xcode upgrade removed /usr/include. That upgrade apparently left the files necessary to build Sage.

comment:3 follow-up: Changed 3 years ago by jhpalmieri

Confirmed: 8.5.beta6 builds but 8.5.rc0 does not. It's the zlib problem. Is there a way to configure Python to detect zlib in the proper location?

comment:4 follow-up: Changed 3 years ago by jhpalmieri

The easiest fix is to delete the file build/pkgs/zlib/spkg-configure.m4, right?

comment:5 in reply to: ↑ 3 Changed 3 years ago by dimpase

Replying to jhpalmieri:

Confirmed: 8.5.beta6 builds but 8.5.rc0 does not. It's the zlib problem. Is there a way to configure Python to detect zlib in the proper location?

Proper, i.e. only known to Xcode? I don't know. It's not really a zlib problem, it's an Apple bug, if you ask me. They really, really want you to do everything via Xcode.

comment:6 in reply to: ↑ 4 Changed 3 years ago by dimpase

Replying to jhpalmieri:

The easiest fix is to delete the file build/pkgs/zlib/spkg-configure.m4, right?

./sage -i zlib

will build zlib from scratch and install it in SAGE_LOCAL, as always.

comment:7 Changed 3 years ago by dimpase

Once again, on the machine with this problem, do you have /usr/include at all?

I guess it might be that you need to run that open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg etc. after you upgraded XCode, as the latter could have left everything in a slightly out of sync state.

If this does not help then we need to find out whether it's that you actually have non-functioning zlib installation before you start building python, or it's a python issue (I suppose you can look at the logs of the python build and see why zlib extension is not built).

comment:8 Changed 3 years ago by jhpalmieri

Once again, no I do not have /usr/include on this machine, and once again, Sage 8.5.beta6 builds on it just fine. The problem is only zlib.

I can't tell whether it's something out of sync with this machine, as you suggest, or whether it could routinely happen to other people. If we think it could affect other machines, which is the more conservative point of view, then we should roll back the zlib part of #26286 to fix it. Assuming it's just my machine is riskier and could mean that others will not be able to build Sage.

comment:9 Changed 3 years ago by dimpase

Rather than blindly rolling back, we'd like to know the cause of the problem, no? This is why I asked you for the log...

comment:10 Changed 3 years ago by vbraun

I don't think we are going to be able to do anything in the short run, for now you just have to install macOS_SDK_headers_for_macOS_10.14

comment:11 Changed 3 years ago by jhpalmieri

From the main config.log:

configure:8396: === checking whether to install the zlib SPKG ===
configure:8410: checking for inflateValidate in -lz
configure:8435: g++ -o conftest -g -O2 -std=gnu++11    conftest.cpp -lz  -lm  >&5
configure:8435: $? = 0
configure:8444: result: yes
configure:8451: checking if zlib is wanted
configure:8473: result: yes
configure:8503: checking for inflateEnd in -lz
configure:8528: gcc -o conftest -g -O2    conftest.c -lz  -lm  >&5
configure:8528: $? = 0
configure:8537: result: yes
configure:8545: checking zlib.h usability
configure:8545: gcc -c -g -O2   conftest.c >&5
configure:8545: $? = 0
configure:8545: result: yes
configure:8545: checking zlib.h presence
configure:8545: gcc -E   conftest.c
configure:8545: $? = 0
configure:8545: result: yes
configure:8545: checking for zlib.h
configure:8545: result: yes

I'm rebuilding now (I deleted the old build to test whether removing build/pkgs/zlib/spkg-configure.m4 would work, and it did). I'll post the Python log when it's available, although I didn't see anything helpful there before, just a comment that the zlib component was not built. I think that Python doesn't know all of the places to look for zlib.h.

comment:12 Changed 3 years ago by jhpalmieri

Volker: a short term fix that doesn't hurt anything is just to remove build/pkgs/zlib/spkg-configure.m4. That is completely safe, so I think we could easily do it for rc1 (if it's decided that it's the right approach).

Changed 3 years ago by jhpalmieri

comment:13 Changed 3 years ago by jhpalmieri

Here is the Python log. I don't see anything helpful in there. By the way, why is Python built twice? Is this a recent thing?

comment:14 follow-up: Changed 3 years ago by dimpase

Thanks. In the log there not even an attempt to build Python's zlib extension, probably meaning that a pre-build check is failing (such as something that checks for headers somewhere...)

By the way, could you check whether zlib extension is built for the Python3? (and if not, how is it failing) There is a sea of difference between Python2 and 3 in regard of how configuration is done, so it could be a bug of sorts in Python2.

Another question is where that zlib.h is found by spkg-configure.m4 Could you search for zlib.h files on the system (I guess, in XCode tree in /Develop*)?

comment:15 Changed 3 years ago by dimpase

In Python2/3 configure.ac, one sees that OSX is treated in a special way:

dnl Check if system zlib has *Copy() functions
dnl
dnl On MacOSX the linker will search for dylibs on the entire linker path
dnl before searching for static libraries. setup.py adds -Wl,-search_paths_firs
t
dnl to revert to a more traditional unix behaviour and make it possible to
dnl override the system libz with a local static library of libz. Temporarily
dnl add that flag to our CFLAGS as well to ensure that we check the version
dnl of libz that will be used by setup.py.
dnl The -L/usr/local/lib is needed as wel to get the same compilation
dnl environment as setup.py (and leaving it out can cause configure to use the
dnl wrong version of the library)
case $ac_sys_system/$ac_sys_release in
Darwin/*)
        _CUR_CFLAGS="${CFLAGS}"
        _CUR_LDFLAGS="${LDFLAGS}"
        CFLAGS="${CFLAGS} -Wl,-search_paths_first"
        LDFLAGS="${LDFLAGS} -Wl,-search_paths_first -L/usr/local/lib"
        ;;
esac

AC_CHECK_LIB(z, inflateCopy, AC_DEFINE(HAVE_ZLIB_COPY, 1, [Define if the zlib library has inflateCopy]))

case $ac_sys_system/$ac_sys_release in
Darwin/*)
        CFLAGS="${_CUR_CFLAGS}"
        LDFLAGS="${_CUR_LDFLAGS}"
        ;;
esac

One may wonder whether it could pick up wrong old zlib from /usr/local, say, and then fail on testing it being recent enough).

By right, Python should have a configure option to tell it where zlib is, and do not fool around the system looking for one...

comment:16 follow-up: Changed 3 years ago by vbraun

Python should be built only once. Our logging concatenates output from different builds.

comment:17 follow-up: Changed 3 years ago by vbraun

IMHO the build/pkgs/zlib/spkg-configure.m4 check should be less leninent, and probably only search zlib in /usr. Right now there is

zlib_places="/usr/local /usr /opt/local /sw"

and lots of our packages are not going to look into /sw for zlib...

comment:18 in reply to: ↑ 14 Changed 3 years ago by jhpalmieri

Replying to dimpase:

Thanks. In the log there not even an attempt to build Python's zlib extension, probably meaning that a pre-build check is failing (such as something that checks for headers somewhere...)

By the way, could you check whether zlib extension is built for the Python3? (and if not, how is it failing)

Same for Python 3. As you note in your later comment, zlib is treated differently on OS X in configure.ac and in Modules/Setup.dist.

Another question is where that zlib.h is found by spkg-configure.m4 Could you search for zlib.h files on the system (I guess, in XCode tree in /Develop*)?

$ locate zlib.h prints the following (along with some matches for files containing "zlib.h", like "nazlib.htf", which I have omitted):

/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/zlib.h

Perhaps the relevant ones are

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks/Kernel.framework/Versions/A/Headers/libkern/zlib.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/zlib.h

In any case, there is nothing relevant in /usr/local. I also don't have /sw or /opt/local/.

comment:19 in reply to: ↑ 16 Changed 3 years ago by jhpalmieri

Replying to vbraun:

Python should be built only once. Our logging concatenates output from different builds.

I did actually know that second part, which is what led me to ask the question in the first place.

If I run make distclean; make, I see this in the Python 2 and 3 log files:

Testing importing of various modules...
ctypes module imported OK
math module imported OK
hashlib module imported OK
crypt module imported OK
readline module imported OK
socket module imported OK
_scproxy module imported OK

real	2m28.119s
user	1m17.674s
sys	0m21.064s
Installing python2-2.7.15.p0
make[4]: warning: -jN forced in submake: disabling jobserver mode.

The "real[TAB][time]" string is typically only printed when the installation is complete. Oh, I see, it's running spkg-build and spkg-install and reporting timings for each. That's a little confusing. The first one should say "Successfully ran spkg-build" at the end.

comment:20 in reply to: ↑ 17 Changed 3 years ago by dimpase

Replying to vbraun:

IMHO the build/pkgs/zlib/spkg-configure.m4 check should be less leninent,

The requirements are needed to make libpg happy, the version we use needs these checks to pass.

and probably only search zlib in /usr.

Homebrew and freebsd have most things in /usr/local.

What is needed for conda?

Right now there is

zlib_places="/usr/local /usr /opt/local /sw"

and lots of our packages are not going to look into /sw for zlib...

comment:21 Changed 3 years ago by dimpase

My current theory is that our macro to recognise/test zlib resorts to calling C complier/linker without extra flags, and everything works, whereas Python's one does something less clever, and fails.

That is we can patch Python's configure.ac and get everything working.

comment:22 follow-ups: Changed 3 years ago by vbraun

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

PS: Conda is not installed at any fixed path, users can place it whereever they want in their home directory.

Last edited 3 years ago by vbraun (previous) (diff)

comment:23 in reply to: ↑ 22 Changed 3 years ago by dimpase

Replying to vbraun:

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

Assuming that mixing system's zlib and Sage's zlib is not a good idea, note that zlib is a dependence (1st or 2nd order) of several other libraries: boost, cbc, glpk, libpng, brial, freetype, giac, libgd, matplotlib, python2/3, etc

that means not using system's zlib forces building of a lot of other Sage libs...

comment:24 follow-up: Changed 3 years ago by vbraun

Since its a common dependency its even more important that we err on the safe side

IMHO either

  1. the user has libraries installed in the standard location /usr, or
  2. the user explicitly configures the libray location, or
  3. we build it ourselves

Trying to poke around in the filesystem in sage configure is just going to be endless hurt as every third-party configure is going to do work a little bit different.

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

comment:25 in reply to: ↑ 24 ; follow-up: Changed 3 years ago by dimpase

  • Cc embray added

Replying to vbraun:

Since its a common dependency its even more important that we err on the safe side

IMHO either

  1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

  1. the user explicitly configures the libray location, or
  1. we build it ourselves

Trying to poke around in the filesystem in sage configure is just going to be endless hurt as every third-party configure is going to do work a little bit different.

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

This is a lot of work for a relatively small crowd, and I don't see why automating it should not be considered.

comment:26 in reply to: ↑ 25 ; follow-up: Changed 3 years ago by vbraun

Replying to dimpase:

  1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

Its pretty much on every unix-like OS, including OSX. The only thing that OSX lacks here is a sane way to distribute software that is not an self-contained opaque application bundle.

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

And the sane way is to let the users explicitly pick their dependencies, not rummage around and see if we can randomly find something that isn't part of the OS/distribution. It should just be a matter of ./configure CPPFLAGS=-I/the/location/include LDFLAGS=-L/the/location/lib

In particular, if you want homebrew integration then make a homebrew package that uses 2. to configure the correct paths.

This is a lot of work for a relatively small crowd, and I don't see why automating it should not be considered.

Surely there are many more homebrew developers than sage developers. Its also going to be much easier to maintain a homebrew package when there is a simple way to configure the sage dependencies, than to fix all possible Sage build pitfalls with whatever broken bits and pieces we may find in the filesystem.

comment:27 in reply to: ↑ 26 Changed 3 years ago by dimpase

Replying to vbraun:

Replying to dimpase:

  1. the user has libraries installed in the standard location /usr, or

it's only (hopefully) standard on Linux...

Its pretty much on every unix-like OS, including OSX.

Not really; on FreeBSD most, but not all, libraries are in /usr/local, rather than in /usr

On Solaris things are just all over the place...

The only thing that OSX lacks here is a sane way to distribute software that is not an self-contained opaque application bundle.

Sanity on OSX is lacking in more than one place, just see above on many incarnations of zlib.h...

we already see a lot of interference from system, and from quasi-distributions' such as Anaconda, libraries on lots of platforms, so instead of fighting this, it's better to let people use Sage on these platforms in a sane way.

And the sane way is to let the users explicitly pick their dependencies, not rummage around and see if we can randomly find something that isn't part of the OS/distribution. It should just be a matter of ./configure CPPFLAGS=-I/the/location/include LDFLAGS=-L/the/location/lib

But /usr/local may be a part of OS/distribution, e.g. Homebrew puts everything in /usr/local, and FreeBSD has most things in /usr/local... Similarly, /sw used to be the place where Fink put all its stuff, IIRC.

And as you say, Conda can be put in /bar/foo, so it would suffice to replace /usr by /bar/foo, not fiddle around with every spkg separately.

Even better, Conda comes with pkg-config, so one could use appropriate autoconf macros, right?

comment:28 follow-up: Changed 3 years ago by jhpalmieri

By the way, I still don't know if this problem is unique to my computer (did something go wrong in an Xcode update?) or if it is a situation shared by others.

comment:29 in reply to: ↑ 28 ; follow-up: Changed 3 years ago by embray

Replying to jhpalmieri:

By the way, I still don't know if this problem is unique to my computer (did something go wrong in an Xcode update?) or if it is a situation shared by others.

Well, it would help to find out if that's the case or not. Is this blocking building releases for OSX? The problem here would be that Sage's configure is detecting some zlib headers on your system, in such a way that Python's configure does not detect the same headers. Why don't you post the config.log for each of them so we can see the specifics, rather than just speculating. Also it's not clear to me what exactly makes you think /usr/include has to do with this. If you have a zlib.h in /usr/local/include it will detect that first, by default. If it isn't found in any of the default prefixes it searches (/usr/local, /usr, /opt/local, /sw) then it shouldn't detect any zlib at all.

In the meantime, you can configure Sage like

./configure --without-zlib

This is clearly counter-intuitive but the --without-zlib flag comes from the AX_CHECK_ZLIB macro and will just prevent it from trying to detect zlib, so it will fail to detect and thus install the spkg. You can give that a try.

Also, I've been meaning to update the SAGE_SPKG_CONFIGURE to add configure flags for explicitly forcing SPGKs/disabling checks for the system lib. But for this case --without-zlib ought already to work.

comment:30 in reply to: ↑ 29 Changed 3 years ago by jhpalmieri

Replying to embray:

The problem here would be that Sage's configure is detecting some zlib headers on your system, in such a way that Python's configure does not detect the same headers. Why don't you post the config.log for each of them so we can see the specifics, rather than just speculating.

Okay, I'll work on that.

Also it's not clear to me what exactly makes you think /usr/include has to do with this. If you have a zlib.h in /usr/local/include it will detect that first, by default. If it isn't found in any of the default prefixes it searches (/usr/local, /usr, /opt/local, /sw) then it shouldn't detect any zlib at all.

See comment:18 for where zlib.h is on my system. I believe that the main configure script is finding a system version of zlib.h, but it's not in a location found by Python.

Changed 3 years ago by jhpalmieri

Changed 3 years ago by jhpalmieri

comment:31 Changed 3 years ago by jhpalmieri

Okay, here is the main config.log and also Python's. Maybe others will find them more illuminating that I do.

comment:32 follow-up: Changed 3 years ago by dimpase

The only part where zlib is mentioned in Python's config log is

configure:11237: checking for inflateCopy in -lz
configure:11262: clang -o conftest -g -O2 -Wl,-search_paths_first  -L/Users/jpalmier/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.6.beta0/local/lib -Wl,-rpath,/Users/jpalmier/Desktop/Sage_stuff/sage_builds/VANILLA/sage-8.6.beta0/local/lib  -Wl,-search_paths_first -L/usr/local/lib conftest.c -lz  -ldl  >&5
configure:11262: $? = 0
configure:11271: result: yes

Strangely, no sign of header check... Also, that -L/usr/local/lib is worrying - don't you by any chance have a copy of zlib in /usr/local ?

And no attempt to build zlib extension in logs/pkg/python2-2.7.15.p0.log.

A python bug? It's really hard to see why zlib extension wasn't even tried to be built...

comment:33 in reply to: ↑ 32 Changed 3 years ago by jhpalmieri

Replying to dimpase:

Also, that -L/usr/local/lib is worrying - don't you by any chance have a copy of zlib in /usr/local ?

No. Not in /usr/local/include, and find /usr/local -name zlib.h -print returns nothing. There's only this:

$ find /usr/local -name zlib* -print
/usr/local/include/boost/iostreams/detail/config/zlib.hpp
/usr/local/include/boost/iostreams/filter/zlib.hpp

Edit: and if I move this stuff out of /usr/local, the main Sage config.log file still says "zlib-1.2.11.p0 not installed (configure check)" and Python 2 is still built without zlib support.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:34 follow-up: Changed 3 years ago by dimpase

dnl Check if system zlib has *Copy() functions
dnl
dnl On MacOSX the linker will search for dylibs on the entire linker path
dnl before searching for static libraries. setup.py adds -Wl,-search_paths_first
dnl to revert to a more traditional unix behaviour and make it possible to
dnl override the system libz with a local static library of libz. Temporarily
dnl add that flag to our CFLAGS as well to ensure that we check the version
dnl of libz that will be used by setup.py. 
dnl The -L/usr/local/lib is needed as wel to get the same compilation 
dnl environment as setup.py (and leaving it out can cause configure to use the
dnl wrong version of the library)
case $ac_sys_system/$ac_sys_release in
Darwin/*) 
	_CUR_CFLAGS="${CFLAGS}"
	_CUR_LDFLAGS="${LDFLAGS}"
	CFLAGS="${CFLAGS} -Wl,-search_paths_first"
	LDFLAGS="${LDFLAGS} -Wl,-search_paths_first -L/usr/local/lib"
	;;
esac

AC_CHECK_LIB(z, inflateCopy, AC_DEFINE(HAVE_ZLIB_COPY, 1, [Define if the zlib library has inflateCopy]))

This is a special casing of Darwin in python2 configure.ac

Can you remove it and try again? (I presume you do have zlib header in /usr/include)

comment:35 Changed 3 years ago by dimpase

If this works then it is an old upstream workaround that should be obsolete with OSX 10.13 and 10.14

comment:36 follow-ups: Changed 3 years ago by vbraun

Python's setup.py checks for the presence (header + library) and version of zlib, and will silently skip zlib if it can't be found. This all happens after the configure run.

        zlib_inc = find_file('zlib.h', [], inc_dirs)
        have_zlib = False
        if zlib_inc is not None:
            zlib_h = zlib_inc[0] + '/zlib.h'
            version = '"0.0.0"'
            version_req = '"1.1.3"'
            if host_platform == 'darwin' and is_macosx_sdk_path(zlib_h):
                zlib_h = os.path.join(macosx_sdk_root(), zlib_h[1:])
            fp = open(zlib_h)
            while 1:
                line = fp.readline()
                if not line:
                    break
                if line.startswith('#define ZLIB_VERSION'):
                    version = line.split()[2]
                    break
            if version >= version_req:
                if (self.compiler.find_library_file(lib_dirs, 'z')):
                    if host_platform == "darwin":
                        zlib_extra_link_args = ('-Wl,-search_paths_first',)
                    else:
                        zlib_extra_link_args = ()
                    exts.append( Extension('zlib', ['zlibmodule.c'],
                                           libraries = ['z'],
                                           extra_link_args = zlib_extra_link_args))
                    have_zlib = True
                else:
                    missing.append('zlib')
            else:
                missing.append('zlib')
        else:
            missing.append('zlib')

comment:37 in reply to: ↑ 34 Changed 3 years ago by jhpalmieri

Replying to dimpase:

(I presume you do have zlib header in /usr/include)

I remind you of the ticket title.

comment:38 in reply to: ↑ 36 Changed 3 years ago by dimpase

Replying to vbraun:

Python's setup.py checks for the presence (header + library) and version of zlib, and will silently skip zlib if it can't be found. This all happens after the configure run. [...]

OK, so the only role of ./configure is to write out the approprite #define in the generated header.

From reading setup.py it looks as if a reasonable attempt is made on OSX to locate the headers in OSX SDK directories, but I cannot guess which of the several checks in this code fragment fails. This needs debugging on OSX 10.14, which I don't have access to. Could someone have a look, just by adding prints in the right places?

comment:39 follow-up: Changed 3 years ago by jhpalmieri

I've added print statements, I don't know if they're in the right places, but as far as I can tell, Python is not looking anywhere in /Applications for zlib.h, and in particular, it is not looking in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/....

Do you have particular requests for where to put print statements, or particular information you would like to see?

comment:40 follow-ups: Changed 3 years ago by jhpalmieri

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages?/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

comment:41 in reply to: ↑ 40 ; follow-up: Changed 3 years ago by dimpase

Replying to jhpalmieri:

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages?/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

I don't see how not installing the headers is normal, unless you want to abandon classic command-line workflow and do everything via Xcode.

The OSX-specific workaround involving adding -L/usr/local/lib to various places seems to indicate that they specifically support, on OSX, libraries installed in /usr/local.

comment:42 in reply to: ↑ 39 ; follow-up: Changed 3 years ago by dimpase

Replying to jhpalmieri:

I've added print statements, I don't know if they're in the right places, but as far as I can tell, Python is not looking anywhere in /Applications for zlib.h, and in particular, it is not looking in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/....

I think is_macosx_sdk_path() expects SDK to be in /System or /Library, but not in /Applications - perhaps this restriction is obsolete, and should be removed?

Do you have particular requests for where to put print statements, or particular information you would like to see?

Does zlib_inc compute as None? What is inc_dirs it is called with? What is the result of macosx_sdk_root() call?

ditto - with is_macosx_sdk_path() hacked to allow /Applications to contain SDK.

comment:43 in reply to: ↑ 42 Changed 3 years ago by jhpalmieri

Replying to dimpase:

Replying to jhpalmieri:

I've added print statements, I don't know if they're in the right places, but as far as I can tell, Python is not looking anywhere in /Applications for zlib.h, and in particular, it is not looking in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/....

I think is_macosx_sdk_path() expects SDK to be in /System or /Library, but not in /Applications - perhaps this restriction is obsolete, and should be removed?

Do you have particular requests for where to put print statements, or particular information you would like to see?

Does zlib_inc compute as None?

Yes.

What is inc_dirs it is called with?

['.', 'Include', './Include', '/usr/local/include', '/Users/jpalmier/Desktop/Python-2.7.15/Include', '/Users/jpalmier/Desktop/Python-2.7.15']

(I'm building in /Users/jpalmier/Desktop/Python-2.7.15.)

What is the result of macosx_sdk_root() call?

'/'

ditto - with is_macosx_sdk_path() hacked to allow /Applications to contain SDK.

No change.

I also tried

CFLAGS="-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" ./configure

which changed the return value of macosx_sdk_root to that directory, but the zlib extension still wasn't bult.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:44 Changed 3 years ago by dimpase

Can you try calling xcrun (?) with appropriate options to get correct SDK root in macos_sdk_root() ?

comment:45 Changed 3 years ago by vbraun

Should be xcrun --show-sdk-path

comment:46 Changed 3 years ago by jhpalmieri

Turns out I had guessed the SDK root correctly: xcrun --show-sdk-path returns

/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

comment:47 Changed 3 years ago by dimpase

that part of setup.py in Python is quite dated, they should really be using xcrun, instead of fooling around with parsing of CFLAGS.

comment:48 in reply to: ↑ 41 Changed 3 years ago by embray

Replying to dimpase:

Replying to jhpalmieri:

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages?/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

I don't see how not installing the headers is normal, unless you want to abandon classic command-line workflow and do everything via Xcode.

Knowing Apple that's probably what they want you to do. Why, after all, would you be using Apple products except if you want to use software written specifically for Apple with Apple's toolchain :)

The OSX-specific workaround involving adding -L/usr/local/lib to various places seems to indicate that they specifically support, on OSX, libraries installed in /usr/local.

You can also set PYTHONCONFIGURE=... to add additional configure flags for configuring Python, if indeed that helps at all.

comment:49 in reply to: ↑ 40 Changed 3 years ago by embray

Replying to jhpalmieri:

I'm also a little puzzled as to why I can't find a Python bug filed for this. Is the standard advice to install /Library/Developer/CommandLineTools/Packages?/macOS_SDK_headers_for_macOS_10.14.pkg or do without zlib support?

It did come up here: https://github.com/pyenv/pyenv/issues/1219

There's also discussion on the Python bug tracker here; it came up in the context of building _tkinter but the issue is similar: https://bugs.python.org/issue34956

With an interesting comment from Ned Deily (emphasis mine):

Python has never fully supported building only from an Xcode-only installation, i.e. we have required at a minimum installing system header files into /usr/include et al by using "xcode-select --install". As of 10.14, xcode-select no longer installs header files in /usr/include. So that further cripples Python builds in that header files for several third-party libraries shipped with macOS are no longer found, like zlib and sqlite3. When using an Xcode-only installation (no Command Line Tools) in previous releases of macOS, I believe it was the case that essentially the system root directory ('/') was searched by the compiler tool chain for header files, for libraries, and for frameworks, and for frameworks the long standard fallback framework search path order was honored by default: first search /Library/Frameworks, then /System/Library/Frameworks. That worked fine if you were installing frameworks like Tcl and TK into /Library/Frameworks and wanting them to override the system ones. If you did not install the Command Line Tools, then the tool chain used the MacOSX.sdk embedded in Xcode.app as the system root for searching. By default, that's found in: /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

Last edited 3 years ago by embray (previous) (diff)

comment:50 Changed 3 years ago by embray

  • Milestone changed from sage-8.6 to sage-8.7

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

comment:51 in reply to: ↑ 22 Changed 3 years ago by dimpase

Replying to vbraun:

Or we can make build/pkgs/zlib/spkg-configure.m4 less clever. The cost for building zlib unnecessary is a few seconds walltime, and the cost for not building zlib when required is total failure.

what we see in comment 11 above is that spkg-configure.m4 already succeeds without even trying to look into any of /usr, /usr/local, etc.

So this is obviously a bug in Python that they resort to that kind of hackery without trying to check whether the compiler/linker already knows where to look.

comment:52 in reply to: ↑ 36 Changed 3 years ago by dimpase

Replying to vbraun:

Python's setup.py checks for the presence (header + library) and version of zlib, and will silently skip zlib if it can't be found. This all happens after the configure run.

        zlib_inc = find_file('zlib.h', [], inc_dirs)
...

digging up shows that in find_file() they special-case OSX, and call macosx_sdk_root() to find the starting point, but the latter does something lame, instead of the latest Xcode magic spell xcrun --show-sdk-path

comment:53 Changed 3 years ago by embray

It sucks because this code has to live, currently, in Python's main source distribution, but something like xcode is such a moving target that it's easy for any special cases made for it can easily become out of sync with Python releases.

For that reason I think it's also perfectly acceptable to patch Python's setup.py in this case, if at least there is a reasonable patch that won't break backwards-compat.

comment:54 Changed 3 years ago by dimpase

xcrun has been there since OSX 10.9, so it's pretty safe to use, at least for SageMath. Note that Python's info on the headers on OSX is outdated. https://devguide.python.org/setup/#macos-and-os-x

Should we open a bug with them?

comment:55 Changed 3 years ago by jhpalmieri

Once again, why don't we just remove build/pkgs/zlib/spkg-configure.m4? Or make the check platform-dependent, so it always builds on OS X >= 10.14. On this computer, building zlib takes 10 seconds, so building it when not necessary is not a big deal.

I like the idea of being able to use system libraries, and if zlib were one of the last pieces, then we should try very hard to fix this. But for now, given a choice between building zlib when it may not be necessary vs. patching Python, I would prefer building zlib.

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:56 follow-up: Changed 3 years ago by dimpase

"just" removing build/pkgs/zlib/spkg-configure.m4 means that anything that depends upon zlib must be built as well.

Besides, I have a draft patch for Python(s) ready (this is diff to CPython master, thus probably needs a rebase for other Pythons)

  • setup.py

    diff --git a/setup.py b/setup.py
    index b96961073b..e1dbd71bb4 100644
    a b def macosx_sdk_root(): 
    134134    cflags = sysconfig.get_config_var('CFLAGS')
    135135    m = re.search(r'-isysroot\s+(\S+)', cflags)
    136136    if m is None:
    137         sysroot = '/'
     137        import subprocess
     138        sysroot = subprocess.check_output(["xcrun", "--show-sdk-path"]).decode("utf-8").strip('\n')
    138139    else:
    139140        sysroot = m.group(1)
    140141    return sysroot
    def is_macosx_sdk_path(path): 
    146147    """
    147148    return ( (path.startswith('/usr/') and not path.startswith('/usr/local'))
    148149                or path.startswith('/System/')
     150                or path.startswith('/Applications/')
    149151                or path.startswith('/Library/') )

This allows building zlib module on CPython master. I guess I'll wrap it into try:...

comment:57 in reply to: ↑ 56 ; follow-up: Changed 3 years ago by jhpalmieri

Replying to dimpase:

"just" removing build/pkgs/zlib/spkg-configure.m4 means that anything that depends upon zlib must be built as well.

I'm curious: how many packages does that affect?

comment:58 in reply to: ↑ 57 Changed 3 years ago by dimpase

Replying to jhpalmieri:

Replying to dimpase:

"just" removing build/pkgs/zlib/spkg-configure.m4 means that anything that depends upon zlib must be built as well.

I'm curious: how many packages does that affect?

there are executables that certainly may use whatever zlib, so apologies for not being precise here, but still this leaves

libpng, iconv, libpng, glpk, cbc, and all of its dependent libraries, such as freetype, libgd, etc.

And it rules out using system-wide Python (this is far from being ready, but in principle not much should prevent one to build Sage in a virtual environment of a system-provided Python).

comment:59 Changed 3 years ago by dimpase

  • Report Upstream changed from N/A to Reported upstream. No feedback yet.

I've opened https://bugs.python.org/issue36231 to discuss this with the upstream.

comment:60 Changed 3 years ago by jhpalmieri

With this patch, Python 2 and 3 both fail to build for me (this is on the machine with no /usr/include and for which I've been having to build Sage's zlib; to test this, I didn't build Sage's zlib).

Python 2:

running build
running build_ext
Traceback (most recent call last):
  File "./setup.py", line 2311, in <module>
    main()
  File "./setup.py", line 2306, in main
    'Lib/smtpd.py']
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/command/build.py", line 127, in run
    self.run_command(cmd_name)
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "./setup.py", line 181, in build_extensions
    missing = self.detect_modules()
  File "./setup.py", line 822, in detect_modules
    search_for_ssl_incs_in
  File "./setup.py", line 80, in find_file
    sysroot = macosx_sdk_root()
  File "./setup.py", line 50, in macosx_sdk_root
    import subprocess
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python2-2.7.15.p0/src/Lib/subprocess.py", line 72, in <module>
    import select
ImportError: No module named select

Python 3:

running build
running build_ext
Traceback (most recent call last):
  File "./setup.py", line 2404, in <module>
    main()
  File "./setup.py", line 2399, in main
    "Tools/scripts/2to3", "Tools/scripts/pyvenv"]
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "./setup.py", line 230, in build_extensions
    missing = self.detect_modules()
  File "./setup.py", line 852, in detect_modules
    search_for_ssl_incs_in
  File "./setup.py", line 126, in find_file
    sysroot = macosx_sdk_root()
  File "./setup.py", line 96, in macosx_sdk_root
    import subprocess
  File "/Users/jpalmier/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.7.beta6/local/var/tmp/sage/build/python3-3.6.6.p0/src/Lib/subprocess.py", line 136, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

The problem seems to be the new line import subprocess in setup.py. I actually don't know how setup.py is supposed to work: how much of Python's functionality can you expect to be working when it is called?

comment:61 Changed 3 years ago by dimpase

Sorry, it's actually not a complete patch. See https://bugs.python.org/issue36231 for a complete one for Python 3 (although it might not cleanly apply to Sage's Python 3, as it's for Python's master).

OK, I must say I misunderstood the issue a bit - that's why on Python bug tracker I posted a patch which simply triggers building of _posixsubprocess and a bunch of other modules (some of them needed for just testing).

Surely using subprocess may be substituted by something available at this point, I just don't know without looking into it more what it should be.

comment:62 Changed 3 years ago by jhpalmieri

Okay, I see. I got Python 3 to build successfully, but when I tried something similar for Python 2, it builds but without the zlib module. (When I say "something similar", I mean that I patched Modules/Setup.dist until import subprocess in setup.py stopped giving errors.) Maybe I'm doing something wrong, and in any case, it's a good sign that it works for Python 3.

comment:63 Changed 3 years ago by vbraun

How likely is it that upstream will fix this for Python 2? My hunch is that they won't, so I would be fine with shipping whatever patch needed to make it work.

comment:64 Changed 3 years ago by embray

  • Milestone changed from sage-8.7 to sage-8.8

Moving all blocker/critical issues from 8.7 to 8.8.

comment:65 Changed 3 years ago by embray

Yep, backporting patches to Python 2.7 for build issues makes perfect sense.

comment:66 Changed 3 years ago by embray

Ned Deily wrote on the BPO issue that he's working on a potential fix but didn't really explain his approach; absent that I can look into it. I think something like Dima's patch would work, but it should be run at ./configure time. Just need to check for the xcrun program and use it if so; if it doesn't exist (e.g. on older OSX) it's fine, because there the issue is not relevant anyways.

comment:67 follow-up: Changed 3 years ago by embray

One thing I'm concerned about is that I don't know exactly how to reproduce this issue, in particular given that the one OSX machine I currently have access to already has a /usr/include/zlib.h for some reason.

comment:68 in reply to: ↑ 67 Changed 3 years ago by dimpase

Replying to embray:

One thing I'm concerned about is that I don't know exactly how to reproduce this issue, in particular given that the one OSX machine I currently have access to already has a /usr/include/zlib.h for some reason.

One can do

sudo mv /usr/include /usr/blah 
sudo mv /usr/local/include /usr/local/blah 

comment:69 Changed 3 years ago by embray

The machine I have access to is one of the Sage buildbots, so I don't want to break anything like that :)

comment:70 Changed 3 years ago by vbraun

  • Branch set to u/vbraun/os_x_mojave_without__usr_include__python_builds_without_zlib_support

comment:71 follow-up: Changed 3 years ago by vbraun

  • Commit set to 3d49fc2eb17db274e4d9a947ea163d740922205b

gfortran still dies with

The directory that should contain system headers does not exist:
  /usr/include

even after deleting build/pkgs/zlib/spkg-configure.m4 (actually I removed the corresponding section from ./configure since autotools are not installed)


New commits:

3d49fc2Delete the zlib spkg-configure.m4
Last edited 3 years ago by vbraun (previous) (diff)

comment:72 Changed 3 years ago by dimpase

huh? just fix it, don't delete it! It calls a crappy macro m4/ax_check_zlib.m4, basically lifted off the net, it can be improved to get rid of an explicit looping through directories.

comment:73 Changed 3 years ago by dimpase

By the way, ​https://bugs.python.org/issue36231 shows that it's perfectly possible to build Python on OSX with zlib from Xcode.

And see comment:58 why using system's zlib is a good idea.

comment:74 Changed 3 years ago by jhpalmieri

Any estimates on when a fix will be available? If it's going to take a while, I suggest using Volker's suggestion for now and then reimplementing build/pkgs/zlib/spkg-configure.m4 when the fix is ready.

comment:75 Changed 3 years ago by dimpase

I only have access to an extremely slow, 8 y.o. OSX machine, and it cannot run 10.14, only 10.13. And I neither use OSX for work nor am paid for porting Python on OSX...

I think Volker’s “fix” must not go through in this form. Make it conditional for this corner case of building for on a particularly bare OSX configuration.

comment:76 in reply to: ↑ 71 ; follow-up: Changed 3 years ago by dimpase

Replying to vbraun:

gfortran still dies with

The directory that should contain system headers does not exist:
  /usr/include

gcc has --with-native-system-header-dir= option (not shown in the output of ./configure -h, but present in docs and in configure itself. One can use already mentioned. xcrun --show-sdk-path call to compute the value to be passed.

comment:77 Changed 3 years ago by dimpase

  • Description modified (diff)
  • Summary changed from OS X Mojave without /usr/include: Python builds without zlib support to OS X Mojave without /usr/include: a range of issues

comment:78 Changed 3 years ago by vbraun

PS: the osx buildbot worker had his /usr/include wiped in the last osx update

comment:79 Changed 3 years ago by dimpase

I'm seting up a remote paid-for (from the grant, hopefully) Mac mini with 10.14, hopefully I'd sort these issues out.

comment:80 in reply to: ↑ 76 ; follow-up: Changed 3 years ago by jhpalmieri

Replying to dimpase:

Replying to vbraun:

gfortran still dies with

The directory that should contain system headers does not exist:
  /usr/include

gcc has --with-native-system-header-dir= option (not shown in the output of ./configure -h, but present in docs and in configure itself. One can use already mentioned. xcrun --show-sdk-path call to compute the value to be passed.

Using GCC_CONFIGURE="--with-sysroot=`xcrun --show-sdk-path` $GCC_CONFIGURE" worked for me. (I didn't see this earlier because I've installed my own gfortran on each of my OS X machines so I don't have to wait for Sage to build it. But when I hid my copy, I get the same error.)

Last edited 3 years ago by jhpalmieri (previous) (diff)

comment:81 in reply to: ↑ 80 Changed 3 years ago by dimpase

Replying to jhpalmieri:

Using GCC_CONFIGURE="--with-sysroot=`xcrun --show-sdk-path` $GCC_CONFIGURE" worked for me. (I didn't see this earlier because I've installed my own gfortran on each of my OS X machines so I don't have to wait for Sage to build it. But when I hid my copy, I get the same error.)

yes, this works (checking on a very bare MacOS 10.14, with nothing except Command Line Tools for Xcode 10.2 installed), thanks:

  • build/pkgs/gcc/build-gcc

    a b if [ "$UNAME" = "Darwin" ]; then 
    2525    # files prior to comparison during bootstrap (broken by Xcode 6.3).
    2626    # See #18156
    2727    GCC_CONFIGURE="--with-build-config=bootstrap-debug $GCC_CONFIGURE"
     28    GCC_CONFIGURE="--with-sysroot=`xcrun --show-sdk-path` $GCC_CONFIGURE"
    2829fi
    2930
    3031# Let Gfortran build on Raspberry Pi using hard floats.

I suppose it should be done conditionally, only if there is no /usr/include present.

comment:82 Changed 3 years ago by vbraun

I've added that to the branch, now gfortran builds...

comment:83 Changed 3 years ago by git

  • Commit changed from 3d49fc2eb17db274e4d9a947ea163d740922205b to 758e699ece926f3bff37ee892ef9729b1c262204

Branch pushed to git repo; I updated commit sha1. New commits:

758e699Add Xcode sdk path

comment:84 Changed 3 years ago by vbraun

  • Status changed from new to needs_review

Passes so I'd suggest we merge this to get the OSX buildbot up and running again. Feel free to reintroduce the zlib configure in a later ticket, then we'll be able to test it against a buildbot without the extra headers installed.

comment:85 follow-up: Changed 3 years ago by dimpase

If you are really in rush we can add a condition disabling spkg-configure.m4 for zlib in the case there is no /usr/include present. Why is this a problem, it's one if block, right?

I hopefully soon will have a Python fix ready that would make this not necessary.

Last edited 3 years ago by dimpase (previous) (diff)

comment:86 Changed 3 years ago by dimpase

  • Reviewers set to Dima Pasechnik

Instead of removing spkg-configure.m4, apply

  • build/pkgs/zlib/spkg-configure.m4

    a b SAGE_SPKG_CONFIGURE([zlib], [ 
    55        AX_CHECK_ZLIB([], [zlib_cv_libz=no])
    66    ])
    77    AS_IF([test "x$zlib_cv_libz" != "xyes"], [sage_spkg_install_zlib=yes])
     8    AC_CHECK_FILE([/usr/include/],[],[sage_spkg_install_zlib=yes])
    89])

this disables the usage of zlib from the system if /usr/include/ is absent. I just tested this on MacOS. With this change, please feel free to give it positive review.

Last edited 3 years ago by dimpase (previous) (diff)

comment:87 Changed 3 years ago by git

  • Commit changed from 758e699ece926f3bff37ee892ef9729b1c262204 to db50d098c3f01147d5c358c77c0e503d5377d8ec

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

25761b7Add Xcode sdk path
db50d09Disable the usage of zlib from the system if /usr/include/ is absent

comment:88 Changed 3 years ago by vbraun

  • Status changed from needs_review to positive_review

You can push commits instead of pasting diffs to the chat, you know ;-)

New commits:

25761b7Add Xcode sdk path
db50d09Disable the usage of zlib from the system if /usr/include/ is absent

comment:89 Changed 3 years ago by vbraun

  • Authors set to Dima Pasechnik, Volker Braun
  • Reviewers changed from Dima Pasechnik to Dima Pasechnik, Volker Braun

comment:90 Changed 3 years ago by vbraun

  • Branch changed from u/vbraun/os_x_mojave_without__usr_include__python_builds_without_zlib_support to db50d098c3f01147d5c358c77c0e503d5377d8ec
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:91 in reply to: ↑ 85 Changed 3 years ago by dimpase

  • Commit db50d098c3f01147d5c358c77c0e503d5377d8ec deleted

Replying to dimpase:

If you are really in rush we can add a condition disabling spkg-configure.m4 for zlib in the case there is no /usr/include present. Why is this a problem, it's one if block, right?

I hopefully soon will have a Python fix ready that would make this not necessary.

Patching Python 3 to do this is easy, Erik's suggestion on https://bugs.python.org/issue36231 really makes it a 1-line addition to Python's configure.ac and 2 more lines to setup.py, but backporting this to Python 2 is not straightforward, as it has more brain-dead setup.py, which seems to requires manual adjustments to a bunch of module configurations.

comment:92 follow-up: Changed 3 years ago by embray

IMO this is kinda weak, albeit harmless for the time being (though the line added to spkg-configure.m4 should at least have had a comment, but at least we can git blame it easily...). I think Python should be patched instead.

comment:93 in reply to: ↑ 92 Changed 3 years ago by dimpase

Replying to embray:

IMO this is kinda weak, albeit harmless for the time being (though the line added to spkg-configure.m4 should at least have had a comment, but at least we can git blame it easily...). I think Python should be patched instead.

to be done on #27631 - almost there...

Note: See TracTickets for help on using tickets.