Opened 17 months ago

Closed 15 months ago

Last modified 8 weeks ago

#26286 closed enhancement (fixed)

more system package checks

Reported by: dimpase Owned by:
Priority: major Milestone: sage-8.5
Component: build Keywords: spkg-configure
Cc: embray Merged in:
Authors: Erik Bray, Dima Pasechnik Reviewers: Erik Bray, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 1d7b12e (Commits) Commit:
Dependencies: #24919, #26660 Stopgaps:

Description (last modified by dimpase)

This is a follow-up to #24919

Some leftovers from there:

  • add amd64 to yasm check
  • simplify some more pre-existing spkg-configure.m4

Already on the branch:

  • xz (+lzma)
  • zlib
  • zeromq

Change History (47)

comment:1 Changed 16 months ago by dimpase

  • Cc embray added
  • Description modified (diff)

comment:2 Changed 16 months ago by dimpase

  • Branch set to u/dimpase/build/t26286
  • Commit set to c00a92cf5b6b9fe5c92e6357a90cabf7c5976a15
  • Description modified (diff)

comment:3 Changed 16 months ago by git

  • Commit changed from c00a92cf5b6b9fe5c92e6357a90cabf7c5976a15 to 1a66f9b2ea112acae22e036ff3987a98ff7f7bf8

Branch pushed to git repo; I updated commit sha1. New commits:

1a66f9brecongnize amd64 as a "good" yasm arch

comment:4 Changed 16 months ago by dimpase

  • Dependencies set to #24919

comment:5 Changed 16 months ago by dimpase

  • Milestone changed from sage-8.4 to sage-8.5

comment:6 Changed 16 months ago by git

  • Commit changed from 1a66f9b2ea112acae22e036ff3987a98ff7f7bf8 to 7d9548bf292f72b864736ad7dcca6cb14cd0dc9d

Branch pushed to git repo; I updated commit sha1. New commits:

7d9548ballow system's zeromq of version at least 4.2.5

comment:7 Changed 16 months ago by git

  • Commit changed from 7d9548bf292f72b864736ad7dcca6cb14cd0dc9d to 73d84c8c24da1e7ed9a22456811a3f882cfd6a8f

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

b4cc8ebUse m4_sinclude instead--this prevents problems when switching between branches where one of these listed files no longer exists
8d01c6eMove these comments inside the macro call to ensure they are actually expanded as part of the macro
cf6367aChanged a bit how SAGE_SPKG_CONFIGURE works:
16c22deImplement an spkg-configure.m4 for patch, demonstrating use of the new system
6526868this check should be handled in gfortran's spkg-configure.m4
a5f783eadd a diagnostic message that is occasionally helpful for understanding the configure output
3bedcd6added better reporting for yasm program+feature check
79de3bdBSD find puts in extra slashes if you leave a trailing slash in the argument
2fc36f0Merge branch 'u/embray/build/autoconf-macros' of trac.sagemath.org:sage into t26660
73d84c8Merge branch 'u/dimpase/build/t26286' of trac.sagemath.org:sage into t26660

comment:8 Changed 16 months ago by dimpase

  • Dependencies changed from #24919 to #24919, #26660

comment:9 Changed 16 months ago by dimpase

  • Description modified (diff)

comment:10 Changed 15 months ago by dimpase

to be fixed:

  • m4/ax_check_liblzma.m4

    a b then 
    111111        CPPFLAGS="$CPPFLAGS -I${LZMA_HOME}/include"
    112112  fi
    113113  AC_LANG_PUSH([C])
    114   AC_CHECK_LIB([lzma], [inflateEnd], [lzma_cv_liblzma=yes], [lzma_cv_liblzma=no])
     114  AC_CHECK_LIB([lzma], [lzma_raw_decoder], [lzma_cv_liblzma=yes], [lzma_cv_liblzma=no])
    115115  AC_CHECK_HEADER([lzma.h], [lzma_cv_lzma_h=yes], [lzma_cv_lzma_h=no])
    116116  AC_LANG_POP([C])
    117117  if test "$lzma_cv_liblzma" = "yes" && test "$lzma_cv_lzma_h" = "yes"

also, if xz test breaks at lzma linking test (but finds its headers), ./configure stops, but it should not...

comment:11 Changed 15 months ago by dimpase

pyzmq's spkg-install needs removal of the line echo "zmq = $SAGE_LOCAL" >> src/setup.cfg, otherwise only Sage's copy of libzmq can be used.


in fact, whenever a package gets spkg-configure.m4, one has to inspect all packages that have it in dependencies for this kind of hardcoded SAGE_LOCAL-only bits in spkg-install etc.

Last edited 15 months ago by dimpase (previous) (diff)

comment:12 Changed 15 months ago by git

  • Commit changed from 73d84c8c24da1e7ed9a22456811a3f882cfd6a8f to 36559cfac0cf7b37dbc91328208bfe8ceb76d30c

Branch pushed to git repo; I updated commit sha1. New commits:

e85c0adMerge branch 'u/dimpase/build/t26286' of trac.sagemath.org:sage into t26286
36559cfadjust pyzmq to use 0mq from outside too, fix lzma check

comment:13 follow-up: Changed 15 months ago by embray

Thank you for working on these. I'm excited to try it on Cygwin

Regarding your comments about zmq that's an interesting point I hadn't thought about. For something like pyzmq I generally assumed that if a system libzmq were used then likely there would be a system pyzmq as well. But it's true that that doesn't have to be the case.

So the question is what to do if some package has as a dependency another package (like libzmq) that was detected at configure-time. If the dependent package can just automatically detect the right location for its dependency, then there is no problem, as I assume is the case for your pyzmq change.

For other packages we might not be so lucky. So we may have to have the configure script output, in some cases, some variables related to dependencies detected at configure-time. For example, if some package really needed to know the correct prefix for libzmq (as opposed to just assuming it's $SAGE_LOCAL) we ought to output some LIBZMQ_PREFIX=... variable to a file that can be sourced by spkg-install. I don't have a precise plan for this, but it is a thought I've had before. This could also be useful for sagelib itself, as we want to be able to tell it exactly where it can find certain runtime dependencies.

comment:14 Changed 15 months ago by embray

Relatedly, I've thought of adding a configure script just for sagelib that repurposes this machinery, but just detects the direct build- and runtime dependencies for Sage itself: This would be useful for packaging and installing just sagelib on systems where all other indirect dependencies are already assumed satisfied. That is, outside the context of sage-the-distribution.

comment:15 in reply to: ↑ 13 Changed 15 months ago by dimpase

Replying to embray:

Thank you for working on these. I'm excited to try it on Cygwin

I'm trying this on Cygwin atm myself - I've managed to get Cygwin running on a Google's cloud Windows 2016 Server. Not sure how close this is to real down to your living room Windows, but still...

Regarding your comments about zmq that's an interesting point I hadn't thought about. For something like pyzmq I generally assumed that if a system libzmq were used then likely there would be a system pyzmq as well. But it's true that that doesn't have to be the case.

This would be OK, sort of, once we can use system's Python, but not now...

So the question is what to do if some package has as a dependency another package (like libzmq) that was detected at configure-time. If the dependent package can just automatically detect the right location for its dependency, then there is no problem, as I assume is the case for your pyzmq change.

For other packages we might not be so lucky. So we may have to have the configure script output, in some cases, some variables related to dependencies detected at configure-time. For example, if some package really needed to know the correct prefix for libzmq (as opposed to just assuming it's $SAGE_LOCAL) we ought to output some LIBZMQ_PREFIX=... variable to a file that can be sourced by spkg-install. I don't have a precise plan for this, but it is a thought I've had before. This could also be useful for sagelib itself, as we want to be able to tell it exactly where it can find certain runtime dependencies.

As we are at that, what's your take on presence of pkg-config on the system? That's of course brings up the question of OSX, where is there no native pkg-config, but with OSX it seems to me that we really must switch over to using Homebrew there, and stop doing the monolithic mega-thing, which is increasingly difficult, cf. e.g. #26713.

comment:16 Changed 15 months ago by embray

Obviously what I'm describing is something like a limited pkg-config. I don't know what the issues are with relying on pkg-config on macos, but I don't think it should be a requirement. There are certainly Python packages that use pkg-config where available to check for dependencies, but there usually end up being fallbacks for mac and windows when pkg-config isn't available.

Whether dependency checks are written in Python, or balled up in a configure script generated by autoconf, those checks would also probably use pkg-config where available, and provide fallbacks where not. Inconvenient and annoying, but a fact of life I suppose :(

comment:17 follow-up: Changed 15 months ago by embray

Perhaps some of our spkg_configure.m4 scripts could even output their own .pc files somewhere in cases where they don't already exist. We would want to have some kind of helper macro for doing that. I'm not sure how much trouble that would be.

comment:18 in reply to: ↑ 17 Changed 15 months ago by dimpase

Replying to embray:

Perhaps some of our spkg_configure.m4 scripts could even output their own .pc files somewhere in cases where they don't already exist. We would want to have some kind of helper macro for doing that. I'm not sure how much trouble that would be.

seems that it's been already something people designed:

https://www.gnu.org/software/autoconf-archive/ax_create_pkgconfig_info.html#ax_create_pkgconfig_info

comment:19 follow-up: Changed 15 months ago by embray

There appears to be a bug in build/pkgs/xz/spkg-configure.m4. On my Linux machine I'm getting:

configure: === checking whether to install the xz SPKG ===
checking if lzma is wanted... yes
checking for inflateEnd in -llzma... no
checking lzma.h usability... yes
checking lzma.h presence... yes
checking for lzma.h... yes
configure: error: either specify a valid lzma installation with --with-lzma=DIR or disable lzma usage with --without-lzma

It appears that my xz version is too low (do we really need it to be a specific version for any known reason)?

Then, it looks like there might be some improper nesting or something, or not fully handling every possible failure case, because I shouldn't get that error.

comment:20 Changed 15 months ago by embray

Looking again at my patch/spkg-configure.m4 I think it also has a bug in how it uses AC_CACHE_CHECK. It is the same "common bug" described on this page: https://www.gnu.org/software/autoconf/manual/autoconf-2.63/html_node/Caching-Results.html

comment:21 follow-up: Changed 15 months ago by embray

Here are the patches I made to fix this, if you want to try applying it your branch:

  • build/pkgs/curl/spkg-configure.m4

    diff --git a/build/pkgs/curl/spkg-configure.m4 b/build/pkgs/curl/spkg-configure.m4
    index 7be9791..e6f72b0 100644
    a b SAGE_SPKG_CONFIGURE([curl], [ 
    22    # Only install curl (and libcurl) if we cannot find version 7.22 or
    33    # later, since that is what R needs.
    44       AC_CACHE_CHECK([for curl 7.22], [ac_cv_path_CURL], [
    5                AC_PATH_PROGS_FEATURE_CHECK([CURL], [curl],
    6                [${ac_path_CURL}-config --checkfor 7.22 >/dev/null 2>/dev/null && ac_cv_path_CURL=${ac_path_CURL}],
    7                [sage_spkg_install_curl=yes; ac_cv_path_CURL=no])
     5               AC_PATH_PROGS_FEATURE_CHECK([CURL], [curl], [
     6                 ${ac_path_CURL}-config --checkfor 7.22 >/dev/null 2>/dev/null && ac_cv_path_CURL=${ac_path_CURL}
     7        ])
    88    ])
     9    AS_IF([test -z "$ac_cv_path_CURL"], [sage_spkg_install_curl=yes])
    910    # If (lib)curl is installed, we need to check for the curl header
    1011    # file, too.
    1112    AS_IF([test $sage_spkg_install_curl = no], [
  • build/pkgs/patch/spkg-configure.m4

    diff --git a/build/pkgs/patch/spkg-configure.m4 b/build/pkgs/patch/spkg-configure.m4
    index f194118..02adb91 100644
    a b SAGE_SPKG_CONFIGURE( 
    88                AX_COMPARE_VERSION([$patch_version], [ge], [2.7.0], [
    99                    ac_cv_path_PATCH="$ac_path_PATCH"
    1010                ])
    11             ])], [sage_spkg_install_patch=yes; ac_cv_path_PATCH=no
     11            ])
    1212        ])
    1313    ])
     14    AS_IF([test -z "$ac_cv_path_PATCH"], [sage_spkg_install_patch=yes])
    1415])
  • build/pkgs/xz/spkg-configure.m4

    diff --git a/build/pkgs/xz/spkg-configure.m4 b/build/pkgs/xz/spkg-configure.m4
    index 1068b6d..7cde0ee 100644
    a b  
    11SAGE_SPKG_CONFIGURE(
    22    [xz], [
    3         AX_CHECK_LZMA([
    4          AC_CACHE_CHECK([for xz >= 5.2.2], [ac_cv_path_XZ], [
     3    AX_CHECK_LZMA([
     4        AC_CACHE_CHECK([for xz >= 5.0.0], [ac_cv_path_XZ], [
    55            AC_PATH_PROGS_FEATURE_CHECK([XZ], [xz], [
    66              xz_version=`$ac_path_XZ --version 2>&1 \
    77                | cut -d' ' -f4 | $SED -n 1p`
    88              AS_IF([test -n "$xz_version"], [
    9                 AX_COMPARE_VERSION([$xz_version], [ge], [5.2.2], [
    10                     ac_cv_path_XZ="$ac_path_XZ"
    11             ])
    12             ])], [sage_spkg_install_xz=yes; ac_cv_path_XZ=no
     9                  AX_COMPARE_VERSION([$xz_version], [ge], [5.0.0], [
     10                      ac_cv_path_XZ="$ac_path_XZ"
     11                  ])
     12              ])
    1313          ])
    14           ])
    15        ])
     14          AS_IF([test -z "$ac_cv_path_XZ"], [sage_spkg_install_xz=yes])
     15      ])
     16    ], [sage_spkg_install_xz=yes])
    1617])
    17 
  • m4/ax_check_liblzma.m4

    diff --git a/m4/ax_check_liblzma.m4 b/m4/ax_check_liblzma.m4
    index 7b3771e..4e7218a 100644
    a b then 
    111111        CPPFLAGS="$CPPFLAGS -I${LZMA_HOME}/include"
    112112  fi
    113113  AC_LANG_PUSH([C])
    114   AC_CHECK_LIB([lzma], [inflateEnd], [lzma_cv_liblzma=yes], [lzma_cv_liblzma=no])
     114  AC_CHECK_LIB([lzma], [lzma_raw_decoder], [lzma_cv_liblzma=yes], [lzma_cv_liblzma=no])
    115115  AC_CHECK_HEADER([lzma.h], [lzma_cv_lzma_h=yes], [lzma_cv_lzma_h=no])
    116116  AC_LANG_POP([C])
    117117  if test "$lzma_cv_liblzma" = "yes" && test "$lzma_cv_lzma_h" = "yes"

This includes the update you mentioned for ax_check_liblzma. I had forgotten that you said you wrote that by basically copying the one for zlib.

I also tried lowering the required version to 5.0.0. I don't know for sure whether or not that's acceptable but I'll try it for now.

comment:22 Changed 15 months ago by embray

For zeromq and zlib we still need to set sage_spkg_install_<spkg-name>=yes when the checks fail.

comment:23 in reply to: ↑ 19 Changed 15 months ago by embray

Replying to embray:

It appears that my xz version is too low (do we really need it to be a specific version for any known reason)?

AFAICT no: it was just the latest version when the xz SPKG was first added to Sage. Unless there is a known problem with some lower version, setting 5.0.0 as a minimum seems fine (if indeed we even need to bother with the version check).

comment:24 follow-up: Changed 15 months ago by embray

Welp, after compiling with my system zlib I get:

ImportError: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /home/embray/src/sagemath/sage/local/lib/libpng16.so.16)

It seems libpng16 has a pretty strong dependency on the ZLIB_1.2.9 property: https://bugs.launchpad.net/ubuntu/+source/libpng1.6/+bug/1780233

comment:25 in reply to: ↑ 24 Changed 15 months ago by dimpase

Replying to embray:

Welp, after compiling with my system zlib I get:

ImportError: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /home/embray/src/sagemath/sage/local/lib/libpng16.so.16)

It seems libpng16 has a pretty strong dependency on the ZLIB_1.2.9 property: https://bugs.launchpad.net/ubuntu/+source/libpng1.6/+bug/1780233

I now recall seeing something along these lines on Fedora, only there this lib is in /lib64/, and it's not checked by the macro. Not sure it's the same problem or a different one.

Last edited 15 months ago by dimpase (previous) (diff)

comment:26 in reply to: ↑ 21 Changed 15 months ago by dimpase

Replying to embray:

Here are the patches I made to fix this, if you want to try applying it your branch:

could you please post them as a branch (perhaps just make a public branch for this ticket), I have trouble applying them.

comment:27 Changed 15 months ago by embray

Sure, I'll just make a public branch.

comment:28 Changed 15 months ago by embray

  • Branch changed from u/dimpase/build/t26286 to public/build/t26286
  • Commit changed from 36559cfac0cf7b37dbc91328208bfe8ceb76d30c to 6845ea672334fbb25014f607215c89f6aae8e3b7

Public branch with my fixes. Still need to look more into the zlib+libpng issue :/


New commits:

2f97065fix how AC_CACHE_CHECK is used for patch and for curl
f976266clean up the spkg-configure.m4 for xz a bit
6845ea6these macros need to set the sage_spkg_install_<pkgname>=yes variable

comment:29 Changed 15 months ago by git

  • Commit changed from 6845ea672334fbb25014f607215c89f6aae8e3b7 to 1d7b12e903f92c43ff49ad9ebfb1c84e978dc625

Branch pushed to git repo; I updated commit sha1. New commits:

1d7b12eensure that our zlib has inflateValidate

comment:30 follow-up: Changed 15 months ago by embray

The libpng requirement for ZLIB_1.2.9 does appear to be quite strict. I haven't checked, but I'm betting that might be way a zlib package was added to Sage in the first place (or possibly the converse if an older zlib was added to Sage first, and later had bad interactions with the system's libpng).

The attached fix works for me but it would be good to get some osx verification.


New commits:

1d7b12eensure that our zlib has inflateValidate

comment:31 in reply to: ↑ 30 Changed 15 months ago by dimpase

Replying to embray:

The libpng requirement for ZLIB_1.2.9 does appear to be quite strict. I haven't checked, but I'm betting that might be way a zlib package was added to Sage in the first place (or possibly the converse if an older zlib was added to Sage first, and later had bad interactions with the system's libpng).

The attached fix works for me but it would be good to get some osx verification.


New commits:

1d7b12eensure that our zlib has inflateValidate

this does work on OSX, at least on 10.13 (which does supply zlib 1.2.11)

comment:32 follow-up: Changed 15 months ago by jhpalmieri

What about bzip2 as another candidate? I don't have time to work on it myself, but it might be easy to add.

comment:33 Changed 15 months ago by dimpase

yes, this should do doable.

But, as well, the more I think about it the more it looks to me that we should seriously consider using Homebrew on OSX, with this sort of approach we won't be having library clashes one sometimes sees with Homebrew...

comment:34 in reply to: ↑ 32 Changed 15 months ago by embray

Replying to jhpalmieri:

What about bzip2 as another candidate? I don't have time to work on it myself, but it might be easy to add.

It's definitely on the list. For the sake of keeping this ticket tractable I don't think I want to add too many additional package checks on this one ticket, but we should definitely have a followup for bz2, and many other packages (ncurses is high on my list because it takes forever to compile on Cygwin for some reason).

comment:35 Changed 15 months ago by dimpase

I'm now trying this branch on OSX/Homebrew, with everything for which we have spkg-configure.m4 coming from either the Xcode or Homebrew; looks promising.

comment:36 Changed 15 months ago by dimpase

OK, here are OSX results of make ptest: two segfaults (not reproducible at sage prompt, must be some weird testing framework problem):

$ ./sage -t --long src/sage/combinat/designs/ext_rep.py
Running doctests with ID 2018-12-05-10-31-31-f39a55d8.
Git branch: pkgcheck
Using --optional=dochtml,mpir,python2,sage
Doctesting 1 file.
sage -t --long --warn-long 113.9 src/sage/combinat/designs/ext_rep.py
    Killed due to abort
**********************************************************************
Tests run before process (pid=24472) failed:
sage: from sage.combinat.designs import ext_rep ## line 478 ##
sage: file_loc = ext_rep.dump_to_tmpfile("boo") ## line 479 ##
sage: os.remove(file_loc) ## line 480 ##
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 481 ##
0
sage: from sage.combinat.designs import ext_rep ## line 497 ##
sage: ext_rep.check_dtrs_protocols('source', '2.0') ## line 498 ##
sage: ext_rep.check_dtrs_protocols('source', '3.0') ## line 499 ##
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 503 ##
0
sage: from sage.combinat.designs import ext_rep ## line 519 ##
sage: file_loc = ext_rep.dump_to_tmpfile(ext_rep.v2_b2_k2_icgsa) ## line 520 ##
sage: proc = ext_rep.XTreeProcessor() ## line 521 ##
sage: f = ext_rep.open_extrep_file(file_loc) ## line 522 ##
sage: proc.parse(f) ## line 523 ##
sage: f.close() ## line 524 ##
sage: os.remove(file_loc) ## line 525 ##
sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 526 ##
0
sage: from sage.combinat.designs import ext_rep ## line 549 ##
sage: file_loc = ext_rep.dump_to_tmpfile(ext_rep.v2_b2_k2_icgsa) ## line 550 ##
sage: proc = ext_rep.XTreeProcessor() ## line 551 ##
sage: s = ext_rep.open_extrep_url("file://" + file_loc) ## line 552 ##
objc[24472]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[24472]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
------------------------------------------------------------------------
0   signals.so                          0x0000000104c5d34a print_backtrace + 58
1   signals.so                          0x0000000104c619a3 sigdie + 67
2   signals.so                          0x0000000104c61900 sigdie_for_sig + 256
3   signals.so                          0x0000000104c617b0 _sig_on_trampoline + 0
4   libsystem_platform.dylib            0x00007fff63caaf5a _sigtramp + 26
5   ???                                 0x3237343432000000 0x0 + 3618418224397287424
6   libsystem_kernel.dylib              0x00007fff63ae625f abort_with_payload_wrapper_internal + 0
7   libobjc.A.dylib                     0x00007fff62d87521 _ZL12_objc_fatalvyyPKcP13__va_list_tag + 108
8   libobjc.A.dylib                     0x00007fff62d873e3 __objc_error + 0
9   libobjc.A.dylib                     0x00007fff62d87f94 _ZL25lockAndFinishInitializingP10objc_classS0_ + 0
10  libobjc.A.dylib                     0x00007fff62d78e8b lookUpImpOrForward + 228
11  libobjc.A.dylib                     0x00007fff62d78914 _objc_msgSend_uncached + 68
12  CoreFoundation                      0x00007fff3b99c9b2 CFDateCreate + 34
13  CoreFoundation                      0x00007fff3b99a29e __CFBinaryPlistCreateObjectFiltered + 1582
14  CoreFoundation                      0x00007fff3b99b8ed __CFBinaryPlistCreateObjectFiltered + 7293
15  CoreFoundation                      0x00007fff3b9815fb __CFTryParseBinaryPlist + 187
16  CoreFoundation                      0x00007fff3b980ebe _CFPropertyListCreateWithData + 190
17  CoreFoundation                      0x00007fff3b980d50 CFPropertyListCreateWithData + 80
18  CoreFoundation                      0x00007fff3b9990cb -[CFPrefsPlistSource handleReply:toRequestNewDataMessage:onConnection:retryCount:error:] + 683
19  CoreFoundation                      0x00007fff3b998df3 __93-[CFPrefsSearchListSource handleReply:toRequestNewDataMessage:onConnection:retryCount:error:]_block_invoke + 147
20  libxpc.dylib                        0x00007fff63cf0935 xpc_array_apply + 57
21  CoreFoundation                      0x00007fff3b998d2b -[CFPrefsSearchListSource handleReply:toRequestNewDataMessage:onConnection:retryCount:error:] + 299
22  CoreFoundation                      0x00007fff3bb1981c __80-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:]_block_invoke_3.139 + 76
23  CoreFoundation                      0x00007fff3bb429d4 -[_CFXPreferences withConnectionForRole:performBlock:] + 36
24  CoreFoundation                      0x00007fff3bb197c4 __80-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:]_block_invoke_2.138 + 116
25  libsystem_trace.dylib               0x00007fff63ccd9f0 _os_activity_initiate_impl + 53
26  CoreFoundation                      0x00007fff3bb19722 __80-[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:]_block_invoke.136 + 114
27  CoreFoundation                      0x00007fff3bb19067 CFPREFERENCES_IS_WAITING_FOR_USER_CFPREFSD + 39
28  CoreFoundation                      0x00007fff3bb192bc -[CFPrefsSearchListSource alreadylocked_generationCountFromListOfSources:count:] + 348
29  CoreFoundation                      0x00007fff3b997416 -[CFPrefsSearchListSource alreadylocked_copyDictionary] + 326
30  CoreFoundation                      0x00007fff3b996f93 -[CFPrefsSearchListSource alreadylocked_copyValueForKey:] + 67
31  CoreFoundation                      0x00007fff3bad6df5 -[CFPrefsSource copyValueForKey:] + 53
32  CoreFoundation                      0x00007fff3bb414a0 __76-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:]_block_invoke + 32
33  CoreFoundation                      0x00007fff3bb1aa49 __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 297
34  CoreFoundation                      0x00007fff3bb1a8b5 -[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 341
35  CoreFoundation                      0x00007fff3bb41443 -[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 131
36  CoreFoundation                      0x00007fff3b99e4e3 CFPreferencesCopyAppValue + 99
37  SystemConfiguration                 0x00007fff47f7b5e1 SCDynamicStoreCopyProxiesWithOptions + 155
38  _scproxy.so                         0x0000000105d9aace get_proxies + 14
39  libpython2.7.dylib                  0x00000001042f9937 PyEval_EvalFrameEx + 22503
40  libpython2.7.dylib                  0x00000001042fe991 fast_function + 337
41  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
42  libpython2.7.dylib                  0x00000001042fe991 fast_function + 337
43  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
44  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
45  libpython2.7.dylib                  0x00000001042702b4 function_call + 340
46  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
47  libpython2.7.dylib                  0x0000000104255b02 instancemethod_call + 178
48  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
49  libpython2.7.dylib                  0x00000001042fe2c2 PyEval_CallObjectWithKeywords + 162
50  libpython2.7.dylib                  0x0000000104253a43 PyInstance_New + 131
51  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
52  libpython2.7.dylib                  0x00000001042f98e9 PyEval_EvalFrameEx + 22425
53  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
54  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
55  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
56  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
57  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
58  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
59  libpython2.7.dylib                  0x00000001042fe991 fast_function + 337
60  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
61  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
62  libpython2.7.dylib                  0x00000001042f43a1 PyEval_EvalFrameEx + 593
63  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
64  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
65  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
66  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
67  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
68  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
69  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
70  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
71  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
72  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
73  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
74  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
75  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
76  libpython2.7.dylib                  0x00000001042702b4 function_call + 340
77  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
78  libpython2.7.dylib                  0x0000000104255b02 instancemethod_call + 178
79  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
80  libpython2.7.dylib                  0x00000001042abce8 slot_tp_call + 168
81  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
82  libpython2.7.dylib                  0x00000001042f98e9 PyEval_EvalFrameEx + 22425
83  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
84  libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
85  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
86  libpython2.7.dylib                  0x00000001042fe991 fast_function + 337
87  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
88  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
89  libpython2.7.dylib                  0x00000001042702b4 function_call + 340
90  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
91  libpython2.7.dylib                  0x0000000104255b02 instancemethod_call + 178
92  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
93  libpython2.7.dylib                  0x00000001042acd4e slot_tp_init + 174
94  libpython2.7.dylib                  0x00000001042a8bb9 type_call + 313
95  libpython2.7.dylib                  0x0000000104245841 PyObject_Call + 97
96  libpython2.7.dylib                  0x00000001042f98e9 PyEval_EvalFrameEx + 22425
97  libpython2.7.dylib                  0x00000001042fe991 fast_function + 337
98  libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
99  libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
100 libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
101 libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
102 libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
103 libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
104 libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
105 libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
106 libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
107 libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
108 libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
109 libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
110 libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
111 libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
112 libpython2.7.dylib                  0x00000001042fe8ad fast_function + 109
113 libpython2.7.dylib                  0x00000001042f96d4 PyEval_EvalFrameEx + 21892
114 libpython2.7.dylib                  0x00000001042f3ef4 PyEval_EvalCodeEx + 2276
115 libpython2.7.dylib                  0x00000001042f3602 PyEval_EvalCode + 34
116 libpython2.7.dylib                  0x000000010432232d PyRun_FileExFlags + 157
117 libpython2.7.dylib                  0x0000000104321dac PyRun_SimpleFileExFlags + 668
118 libpython2.7.dylib                  0x00000001043386cf Py_Main + 3279
119 libdyld.dylib                       0x00007fff6399c015 start + 1
------------------------------------------------------------------------
Unhandled SIGABRT: An abort() occurred.
This probably occurred because a *compiled* module has a bug
in it and is not properly wrapped with sig_on(), sig_off().
Python will now terminate.
------------------------------------------------------------------------

**********************************************************************
----------------------------------------------------------------------
sage -t --long --warn-long 113.9 src/sage/combinat/designs/ext_rep.py  # Killed due to abort
----------------------------------------------------------------------
Total time for all tests: 0.2 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds

and a very similar one in src/sage/doctest/external.py. And few timeouts - not surprising for an 8-year old machine...

Looks good enough for a positive OSX review to me.

comment:37 follow-up: Changed 15 months ago by embray

This looks like it's still a problem. The reason you're not seeing it at the prompt is that it has something to do with forked processes, which is what you have (by default) in the doctest runner, but not at the command line. So one of these system libraries (perhaps, I'm just guessing) has broken post-fork() behavior.

But I can't see clearly through all that CoreFoundation stuff to know what's going on. There's even something in that stacktrace mentioning cloudConfigurationURL which terrifies me. Apple... :<

comment:38 in reply to: ↑ 37 ; follow-up: Changed 15 months ago by dimpase

Replying to embray:

This looks like it's still a problem. The reason you're not seeing it at the prompt is that it has something to do with forked processes, which is what you have (by default) in the doctest runner, but not at the command line. So one of these system libraries (perhaps, I'm just guessing) has broken post-fork() behavior.

this looks like export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES might be needed, and this has nothing to do with Homebrew or Sage...

We have seen this stuff already, I just already forgot (as one does with nightmares :-))

But I can't see clearly through all that CoreFoundation stuff to know what's going on. There's even something in that stacktrace mentioning cloudConfigurationURL which terrifies me. Apple... :<

comment:39 in reply to: ↑ 38 Changed 15 months ago by embray

Replying to dimpase:

Replying to embray:

This looks like it's still a problem. The reason you're not seeing it at the prompt is that it has something to do with forked processes, which is what you have (by default) in the doctest runner, but not at the command line. So one of these system libraries (perhaps, I'm just guessing) has broken post-fork() behavior.

this looks like export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES might be needed, and this has nothing to do with Homebrew or Sage...

We have seen this stuff already, I just already forgot (as one does with nightmares :-))

I thought maybe it had something to do with that too, but I wasn't sure if you re-ran those tests before/after this patch.

comment:40 Changed 15 months ago by dimpase

  • Authors set to Erik Bray, Dima Pasechnik
  • Reviewers set to Erik Bray, Dima Pasechnik
  • Status changed from new to needs_review

OK, indeed, export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES makes tests work. Should we send this to the bots now?

comment:41 Changed 15 months ago by embray

  • Status changed from needs_review to positive_review

comment:42 Changed 15 months ago by embray

I won't be surprised if the buildbots discover new problems with this but that's what they're there for. Good for me until and unless we hear otherwise.

comment:43 Changed 15 months ago by jhpalmieri

Those OS X failures involving OBJC_DISABLE_... are discussed at #25921.

comment:44 Changed 15 months ago by vbraun

  • Branch changed from public/build/t26286 to 1d7b12e903f92c43ff49ad9ebfb1c84e978dc625
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:45 Changed 14 months ago by jhpalmieri

  • Commit 1d7b12e903f92c43ff49ad9ebfb1c84e978dc625 deleted

This broke the build on an OS X Mojave machine. With Mojave, it is possible to have no directory /usr/include, for example, but with this branch, ./configure found zlib somewhere. The problem is that Python doesn't find it, so Python is built without zlib support, and that breaks the pip build, for example.

I've opened #26899 for this.

comment:46 Changed 12 months ago by dimpase

  • Keywords spkg-configure added

comment:47 Changed 8 weeks ago by dimpase

opened #28906 to deal with potentially missing zlib.pc.

Note: See TracTickets for help on using tickets.