Make Sage build with clang (3.7+) and make it the default on OS X

[ohanar]

Apple stopped supporting GCC in favour of clang. This is to track the status of building Sage with clang with the aim of making it the default compiler used on OS X and other OS where it is the default compiler.

---

Packages failing their tests

• eclib, reported upstream at ​https://github.com/JohnCremona/eclib/issues/19
• giac - patch to pass tests with gcc/pari-2.8+ causes failures with clang fixed by #24174
• ccache #22836

---

Supporting clang not only on OSX:

• porting to clang/FreeBSD -- #22679

[jdemeyer]

I guess many of these tickets are useful for porting in general.

[leif]

Added further ticket references.

[leif]

Description presumably needs some updating...

[leif]

eclib has been fixed upstream last year (IIRC) as well; currently re-checking whether our current version still builds...

[leif]

So on Darwin at least, we'd still have to build gfortran... (i.e., GCC with only the Fortran frontend and libgfortran.)

Or resort to f2c again... :-)

[leif]

Replying to leif:

eclib has been fixed upstream last year (IIRC) as well; currently re-checking whether our current version still builds...

(Still) works for me, clang 3.2; make check (and some doctests) passed.

[leif]

I think the upcoming GCC 4.9 will help us with Clang as well... ;-)

[leif]

Checking current status with Sage 6.2 and clang 3.4.1 ...

[leif]

Replying to leif:

Checking current status with Sage 6.2 and clang 3.4.1 ...

The following package(s) may have failed to build:

package: atlas-3.10.1.20140210

package: boehm_gc-7.2d.p0

package: elliptic_curves-0.7

package: gf2x-1.1.p0

package: gfan-0.5.p0

package: lcalc-1.23.p12

package: libgap-4.7.4

package: polybori-0.8.3

package: pynac-0.3.2

package: ratpoints-2.1.3.p3

package: symmetrica-2.0.p9

[More to come.]

[leif]

Goes on with

package: eclib-20140128

package: linbox-1.3.2.p0

package: numpy-1.7.0

package: singular-3.1.6.p1

[leif]

Replying to leif:

package: elliptic_curves-0.7

No idea what's going on there (ImportError: No module named _sqlite3); just recompiling sqlite with GCC doesn't help.

[leif]

Many of the hundreds of C++ errors are of the form

error: friend declaration specifying a default argument must be the only declaration
error: friend declaration specifying a default argument must be a definition

[leif]

Replying to leif:

Replying to leif:

package: elliptic_curves-0.7

No idea what's going on there (ImportError: No module named _sqlite3); just recompiling sqlite with GCC doesn't help.

Ah, that's a nice one: Python fails to build the _sqlite3 extension because of

clang: error: unknown argument: '-R/path/to/sage-6.2-clang-3.4.1/local/lib'

[leif]

Linbox (without the errors caused by Givaro headers):

../../linbox/solutions/solve.h:314:9: error: no matching conversion for functional-style cast from 'const char [19]' to 'LinBox::LinBoxFailure'

[leif]

NumPy fails to build due to (at least) various problems with xmmintrin.h.

[leif]

Ok, removed gf2x and NumPy again; this was my fault, as clang kept including the wrong xmmintrin.h header (i.e., GCC's instead of its own).

[leif]

For Python's distutils, a patch is attached to #16376 in order to not pass -R (but -Wl,-R,...) to clang, such that e.g. the _sqlite3 module also builds with clang.

[leif]

By the way, R 3.0.2 built for me with clang 3.4.1 without any problems (using gfortran 4.9.0).

[leif]

Someone^TM should update the list w.r.t. more recent versions of Sage and clang...

[fbissey]

I may post a branch here for stuff that I don't feel are in the right shape for their own ticket.

[rws]

Just to document the problem I had after applying all the patches above. I've given up for now:

[libgap-4.8.3] /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux
/bin/ld: .libs/libgap_la-ariths.o: relocation R_X86_64_32S against `libGAP_ZEROOp
' can not be used when making a shared object; recompile with -fPIC
[libgap-4.8.3] .libs/libgap_la-ariths.o: error adding symbols: Bad value
[libgap-4.8.3] clang-3.7: error: linker command failed with exit code 1 (use -v t
o see invocation)
[libgap-4.8.3] Makefile:503: recipe for target 'libgap.la' failed
[libgap-4.8.3] make[5]: *** [libgap.la] Error 1

OpenSuSE 42.1, clang-3.7, gcc 4.8/5/6 in system, invoked with make distclean; CC=clang CXX=clang++ make

[dimpase]

updated the versions we target to more realistic (Xcode 6.4 was based on 3.6.0svn), and meanwhile we are much further, see see ​https://gist.github.com/yamaya/2924292

[fbissey]

I am currently working with xcode 8.0 which is clang 3.8.0. fplll-5.0.x compiles but I haven't checked singular 4. No way to check fpylll without sage building first. I'll post some details on the current problems with sagelib tomorrow. That will be interesting :)

Hint stuff outside sagelib may need fixes.

[fbissey]

So the first failure in building sagelib occurs with sage/finance/time_series.pyx

[sagelib-7.4.rc0] [  1/386] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.o -fno-strict-aliasing
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.c:317:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1777:
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
[sagelib-7.4.rc0] #warning "Using deprecated NumPy API, disable it by " \
[sagelib-7.4.rc0]  ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.c:15252:3: error: non-void function '__pyx_pf_4sage_7finance_11time_series_10TimeSeries_106numpy' should return a value [-Wreturn-type]
[sagelib-7.4.rc0]   import_array();
[sagelib-7.4.rc0]   ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1532:144: note: expanded from macro 'import_array'
[sagelib-7.4.rc0] #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }
[sagelib-7.4.rc0]                                                                                                                                                ^
[sagelib-7.4.rc0] 1 warning and 1 error generated.

The problem is the import_array statement from numpy, more precisely this bit of code from local/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h

#if PY_VERSION_HEX >= 0x03000000
#define NUMPY_IMPORT_ARRAY_RETVAL NULL
#else
#define NUMPY_IMPORT_ARRAY_RETVAL
#endif

#define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }

As you can see for python < 3 NUMPY_IMPORT_ARRAY_RETVAL is defined but empty, therefore the return statement in import_array is empty. This is completely and mainly rely on gcc to be permissive. NULL is not necessarily a good return value either, since it should be an error code. The whole thing assume or impose something on the part of the function calling import_array that is not true in time_series an I am not sure can be imposed in cython (don't know enough about cython).

A quick and dirty solution is to edit _multiarray.py and make NUMPY_IMPORT_ARRAY_RETVAL NULL always. That get us to the next problem.

[fbissey]

Next failure

[sagelib-7.4.rc0] [  1/373] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.o -fno-strict-aliasing
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1334:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_compute_out_neighborhood_cardinality)(struct __pyx_obj_4sage_6graphs_20graph_decompositions_12fast_digraph_FastDigraph *, int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1335:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_popcount32)(int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] 2 errors generated.

I am not sure what to do with that one. I'll just note line 257 is the one enabled by __GNUC__

#ifndef CYTHON_INLINE
  #if defined(__GNUC__)
    #define CYTHON_INLINE __inline__
  #elif defined(_MSC_VER)
    #define CYTHON_INLINE __inline
  #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L
    #define CYTHON_INLINE inline
  #else
    #define CYTHON_INLINE
  #endif
#endif

So clang pretending to be gcc means that CYTHON_INLINE is defined. Of course clang support the GNU form, but it doesn't support a declaration to be inlined as it is in that code. This is clearly a cython code generation issue.

[fbissey]

Hand editing the generated cython code allows us to go a bit further until exactly the same error occurs in another file

[sagelib-7.4.rc0] [  1/363] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.o -fno-strict-aliasing
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:1785:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE uint32_t *(*__pyx_f_4sage_6graphs_4base_19static_sparse_graph_has_edge)(__pyx_t_4sage_6graphs_4base_19static_sparse_graph_short_digraph_s *, int, int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:258:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:1786:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE PyObject *(*__pyx_f_4sage_6graphs_4base_19static_sparse_graph_edge_label)(__pyx_t_4sage_6graphs_4base_19static_sparse_graph_short_digraph_s *, uint32_t *); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:258:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:7758:31: warning: comparison of array '__pyx_v_self->g_rev' not equal to a null pointer is always true [-Wtautological-pointer-compare]
[sagelib-7.4.rc0]   __pyx_t_1 = ((__pyx_v_self->g_rev != NULL) != 0);
[sagelib-7.4.rc0]                 ~~~~~~~~~~~~~~^~~~~    ~~~~
[sagelib-7.4.rc0] 1 warning and 2 errors generated.

That's where I am at right now.

[fbissey]

And then sage/graphs/graph_decompositions/cutwidth.pyx, sage/libs/mpmath/ext_main.pyx, sage/libs/mpmath/ext_libmp.pyx. Next, my modification of __multiarray_api.h is bitting me.

[sagelib-7.4.rc0] [ 43/291] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.o -fno-strict-aliasing
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.c:309:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1777:
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
[sagelib-7.4.rc0] #warning "Using deprecated NumPy API, disable it by " \
[sagelib-7.4.rc0]  ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.c:28680:3: error: void function 'initmatrix_double_dense' should not return a value [-Wreturn-type]
[sagelib-7.4.rc0]   import_array();
[sagelib-7.4.rc0]   ^~~~~~~~~~~~~~
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1532:144: note: expanded from macro 'import_array'
[sagelib-7.4.rc0] #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }
[sagelib-7.4.rc0]                                                                                                                                                ^      ~~~~~~~~~~~~~~~~~~~~~~~~~
[sagelib-7.4.rc0] 1 warning and 1 error generated.
[sagelib-7.4.rc0] error: command 'clang' failed with exit status 1

May be I'll learn to do the correct thing for the first instance.

[fbissey]

????????

[sagelib-7.4.rc0] [  4/249] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_integer_dense.o -U_LB_DEBUG -DDISABLE_COMMENTATOR -I/Users/fbissey/build/sage/local/include/linbox -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -msse4.1 -mfma -mavx2 -DFFLAS_COMPILED -DFFPACK_COMPILED -I/Users/fbissey/build/sage/local//include -fPIC -I/Users/fbissey/build/sage/local/include -std=gnu++11 -std=c99 -I/Users/fbissey/build/sage/local/include/m4ri -mmmx -msse -msse2 -msse3 -fno-strict-aliasing
[sagelib-7.4.rc0] error: invalid argument '-std=c99' not allowed with 'C++/ObjC++'

Now there's a bug somewhere: -std=gnu++11 -std=c99.

[fbissey]

Not sure where linbox is pulled from, but that made it .cpp instead of .c. linbox's .pc file brings the -std=gnu++11, module_list.py brings -std=c99. I can remove it from module_list.py but first I'd like to understand where linbox was pulled from.

[jdemeyer]

Replying to fbissey:

Next failure

[sagelib-7.4.rc0] [  1/373] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.o -fno-strict-aliasing
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1334:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_compute_out_neighborhood_cardinality)(struct __pyx_obj_4sage_6graphs_20graph_decompositions_12fast_digraph_FastDigraph *, int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1335:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_popcount32)(int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] 2 errors generated.

That's a genuine problem with the Sage code. GCC only warns for this (which is actually surprising, since the code makes no sense). Ideally, Cython should already refuse this code instead of generating invalid C.

The fix is easy: remove inline from all declarations in .pxd files.

[fbissey]

Thanks for the pointer Jeroen, that makes two new tickets where I have an idea of what to do. The matrix_integer_dense is a real problem with gcc too

[206/450] x86_64-pc-linux-gnu-g++ -O1 -march=native -pipe -ggdb -fPIC -I/usr/lib64/python2.7/site-packages/cysignals -I/usr/include/openblas -I/usr/include -I/usr/include/python2.7 -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7 -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/sage/ext -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/ext -I/usr/include/python2.7 -c /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/matrix/matrix_integer_dense.cpp -o /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/matrix/matrix_integer_dense.o -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -DOPENBLAS_ARCH_X86_64=1 -DOPENBLAS___64BIT__=1 -O1 -march=native -pipe -ggdb -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -fabi-version=6 -fopenmp -DFFLAS_COMPILED -DFFPACK_COMPILED -O1 -march=native -pipe -ggdb -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mfpmath=sse -fabi-version=6 -I/usr/include/linbox -I/usr/include/openblas -std=c99 -I/usr/include/m4ri -fno-strict-aliasing

Also here it uses g++ rather than gcc. If a .cpp file is generated we should just adjust the options.

[fbissey]

OK I figured out what is pulled from where now. Lots of declarations directly in matrix_integer_dense.pyx rather than the matching .pxd file. sage/arith/multi_modular.pxd is the only pxd file with a distutils declaration -std=c99 and it is pulled in matrix_integer_dense.pyx by

from sage.arith.multi_modular cimport MultiModularBasis

on line 88. linbox is pulled by

######### linbox interface ##########
from sage.libs.linbox.linbox cimport Linbox_integer_dense
cdef Linbox_integer_dense linbox = Linbox_integer_dense()
USE_LINBOX_POLY = True

line 122 to 125.

In the short term I will forego the distutils declaration in multi_modular.pxd and see what happens combined with some changes to module_list.py.

[fbissey]

I will open a new ticket about language selection for various pxd and pxi files, I updated the cython inline tickets with new files as I found them. I am now hitting a problem with pynac for which I'll open a new and cc Ralph ASAP.

[sagelib-7.4.rc1] [1/9] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p3/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.o -std=c++11 -fno-strict-aliasing
[sagelib-7.4.rc1] In file included from /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:336:
[sagelib-7.4.rc1] In file included from /Users/fbissey/build/sage/src/sage/symbolic/ginac_wrap.h:11:
[sagelib-7.4.rc1] In file included from /Users/fbissey/build/sage/local/include/pynac/ginac.h:38:
[sagelib-7.4.rc1] /Users/fbissey/build/sage/local/include/pynac/integral.h:44:5: warning: 'GiNaC::integral::evalf' hides overloaded virtual function [-Woverloaded-virtual]
[sagelib-7.4.rc1]         ex evalf(int level=0) const;
[sagelib-7.4.rc1]            ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/local/include/pynac/basic.h:176:13: note: hidden overloaded virtual function 'GiNaC::basic::evalf' declared here: different number of parameters (2 vs 1)
[sagelib-7.4.rc1]         virtual ex evalf(int level = 0, PyObject* parent=nullptr) const;
[sagelib-7.4.rc1]                    ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:27231:30: error: no member named 'operator!=' in 'std::__1::__list_const_iterator<GiNaC::ex, void *>'
[sagelib-7.4.rc1]     __pyx_t_2 = (__pyx_v_itr.operator!=(__pyx_v_lstend) != 0);
[sagelib-7.4.rc1]                  ~~~~~~~~~~~ ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:27446:30: error: no member named 'operator!=' in 'std::__1::__list_const_iterator<GiNaC::ex, void *>'
[sagelib-7.4.rc1]     __pyx_t_3 = (__pyx_v_itr.operator!=(__pyx_v_found.end()) != 0);
[sagelib-7.4.rc1]                  ~~~~~~~~~~~ ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:28649:30: error: no member named 'operator!=' in 'std::__1::__tree_const_iterator<GiNaC::ex, std::__1::__tree_node<GiNaC::ex, void *> *, long>'
[sagelib-7.4.rc1]     __pyx_t_3 = (__pyx_v_itr.operator!=(__pyx_v_sym_set.end()) != 0);
[sagelib-7.4.rc1]                  ~~~~~~~~~~~ ^
[sagelib-7.4.rc1] 1 warning and 3 errors generated.

Notice that I am down to 9 cythonized file to compile :)

[jdemeyer]

Replying to fbissey:

OK I figured out what is pulled from where now. Lots of declarations directly in matrix_integer_dense.pyx rather than the matching .pxd file. sage/arith/multi_modular.pxd is the only pxd file with a distutils declaration -std=c99

This was a mistake introduced by #17766. The -std=c99 declaration should have been put in the .pyx file, not the .pxd file.

If you make a ticket moving this declaration to multi_modular.pyx, you can set it to positive_review on my behalf.

[jdemeyer]

See also my question at ​https://trac.sagemath.org/ticket/17766#comment:7 :-)

[jdemeyer]

The -std=c99 declaration itself was added in #10281 (yes, I like software archaeology).

[fbissey]

Thanks for pointers Jeroen. What about # distutils: language = ... I also change/added/removed a couple of those. They should also belong to .pyx I presume? That would make things more tidy than the mess I see right now.

[fbissey]

Actually I didn't play as much with language than I remember. I changed only matrix_modn_sparse.pxd from c to c++. I also tried expression.pxd for #21701 but that didn't help - so I won't consider it at this stage. But I think there may a number of .pyx declared c language which become c++ (and .cpp) by inheritance of something in c++ like matrix_integer_dense.pyx.

[jdemeyer]

Replying to fbissey:

Thanks for pointers Jeroen. What about # distutils: language = ... I also change/added/removed a couple of those. They should also belong to .pyx I presume?

Not in general. These things have to be considered on a case-by-case basis.

The rule is: the .pxd files should contain the # distutils directives which are needed for the included (cdef extern from ...) files to compile. In src/sage/arith/multi_modular.pxd, there are no included files (it is only Cython code), so there is no reason to add -std=c99. However, it could be that -std=c99 or language = c++ is needed for a particular header file which is used in a .pxd file. In that case, the # distutils directive must be in the .pxd file.

[jdemeyer]

Replying to fbissey:

I changed only matrix_modn_sparse.pxd from c to c++.

That is certainly a mistake. There is no reason why it should be c++. I would remove the line # distutils: language = c completely, since I don't see the point of it.

[jdemeyer]

Looking at some # distutils directives, I see many mistakes...

It would indeed be good to clean this up.

[fbissey]

Replying to jdemeyer:

Replying to fbissey:

I changed only matrix_modn_sparse.pxd from c to c++.

That is certainly a mistake. There is no reason why it should be c++. I would remove the line # distutils: language = c completely, since I don't see the point of it.

OK will try that. I guess a separate ticket for clean up may help slightly with other stuff.

[jdemeyer]

An example where language = c++ in a .pxd file makes sense is given by Singular: #17254 added # distutils: language = c++ to src/sage/libs/singular/decl.pxd. I think this is correct.

[jdemeyer]

Replying to jdemeyer:

Looking at some # distutils directives, I see many mistakes...

It would indeed be good to clean this up.

I just created #21749 for this, assuming that you haven't done anything yet.

[dimpase]

I've been playing with building Sage with clang 3.9 on Linux (gentoo), and it almost works. The only package that fails in a way I can't seem to be able to fix on the spot is pyzmq (see my comments at the end of ​https://github.com/zeromq/pyzmq/issues/468), and to build sagelib I only need

diff --git a/build/pkgs/libgap/spkg-install b/build/pkgs/libgap/spkg-install
index dd6be37..3a167c9 100755
--- a/build/pkgs/libgap/spkg-install
+++ b/build/pkgs/libgap/spkg-install
@@ -41,6 +41,8 @@ for patch in ../patches/*.patch; do
     fi
 done
 
+CFLAGS="$CFLAGS -fPIC"
+
 echo "Configuring libGAP..."
 ./configure --disable-static \
     CFLAGS="$CFLAGS" CXXFLAGS="$CXXFLAGS" CPPFLAGS="$CPPFLAGS" \
diff --git a/src/sage/libs/m4ri.pxd b/src/sage/libs/m4ri.pxd
index a1fe594..ea07121 100644
--- a/src/sage/libs/m4ri.pxd
+++ b/src/sage/libs/m4ri.pxd
@@ -1,4 +1,3 @@
-# distutils: extra_compile_args = -std=c99
 
 cdef extern from "m4ri/m4ri.h":
     ctypedef int rci_t

After this, I can create a working sage instance by doing ./sage --pip install ipython, which pulls pyzmq-16.0.2, and all gets installed nicely. (there is too much clever AI behind compiler recongnition in Sage's instance of pyzmq - perhaps we should upgrade?)

Running tests now (using ./sage -tp)

Note that pynac fix I proposed on #21701 is not needed here, so it's just an older clang on OSX, it seems.

[fbissey]

Not necessarily. Did you build clang to use clang's version of libstdc++ or are you still using gcc's libstdc++. There is no issue so long as you are using gcc's libstdc++.

[fbissey]

And you need clang-9999 to get the right dependencies to default-libcxx which is llvm's own libstdc++. So I'll assume you are not in the right position to compare with OS X.

[dimpase]

Replying to fbissey:

Not necessarily. Did you build clang to use clang's version of libstdc++ or are you still using gcc's libstdc++. There is no issue so long as you are using gcc's libstdc++.

hmm, I did emerge --ask clang and it pulled in few other things too.

[dimpase]

Replying to fbissey:

And you need clang-9999 to get the right dependencies to default-libcxx which is llvm's own libstdc++. So I'll assume you are not in the right position to compare with OS X.

You are right, it still uses gcc's libs. Is there a clean gentoo way to build such a toolchain? All I get is

# emerge --ask clang-9999
!!! 'clang-9999' is not a valid package atom.
!!! Please check ebuild(5) for full details.

PS - OK, I see, it should have been emerge --ask clang-9999...

[fbissey]

The 9999 denotes a hot version (from git master). There appears to be ebuilds for 3.9.0 (libcxx and libcxxabi) and you should be able to emerge those but you will have to pass some extra arguments to use it instead of gcc's version.

Emerging a 9999 is always a bit tricky. Only do it if you are comfortable with emerge. and for what you did the correct syntax is

emerge =clang-9999

anything including a version number needs "=".

[fbissey]

Replying to dimpase:

I've been playing with building Sage with clang 3.9 on Linux (gentoo), and it almost works. The only package that fails in a way I can't seem to be able to fix on the spot is pyzmq (see my comments at the end of ​https://github.com/zeromq/pyzmq/issues/468), and to build sagelib I only need

diff --git a/build/pkgs/libgap/spkg-install b/build/pkgs/libgap/spkg-install
index dd6be37..3a167c9 100755
--- a/build/pkgs/libgap/spkg-install
+++ b/build/pkgs/libgap/spkg-install
@@ -41,6 +41,8 @@ for patch in ../patches/*.patch; do
     fi
 done
 
+CFLAGS="$CFLAGS -fPIC"
+
 echo "Configuring libGAP..."
 ./configure --disable-static \
     CFLAGS="$CFLAGS" CXXFLAGS="$CXXFLAGS" CPPFLAGS="$CPPFLAGS" \
diff --git a/src/sage/libs/m4ri.pxd b/src/sage/libs/m4ri.pxd
index a1fe594..ea07121 100644
--- a/src/sage/libs/m4ri.pxd
+++ b/src/sage/libs/m4ri.pxd
@@ -1,4 +1,3 @@
-# distutils: extra_compile_args = -std=c99
 
 cdef extern from "m4ri/m4ri.h":
     ctypedef int rci_t

After this, I can create a working sage instance by doing ./sage --pip install ipython, which pulls pyzmq-16.0.2, and all gets installed nicely. (there is too much clever AI behind compiler recongnition in Sage's instance of pyzmq - perhaps we should upgrade?)

Running tests now (using ./sage -tp)

Note that pynac fix I proposed on #21701 is not needed here, so it's just an older clang on OSX, it seems.

Hum, Jeroen and I have just added that (m4ri.pxd) and I think it adds a genuine conflict. We will have to examine this carefully.

[fbissey]

OK, was your problem

[sagelib-7.5.beta3] [ 1/59] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p3/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.o -std=c99 -I/Users/fbissey/build/sage/local/include/m4ri -I/Users/fbissey/build/sage/local/include -g -fPIC -Wall -O2 -mmmx -msse -msse2 -msse3 -fno-strict-aliasing
[sagelib-7.5.beta3] error: invalid argument '-std=c99' not allowed with 'C++/ObjC++'

Dima?

[fbissey]

It may actually be a case of overlinking m4ri is called in more place than it should. sage-on-gentoo compiled with --as-needed only report the following users of m4ri

readelf -d `find /usr/lib64/python2.7/site-packages/sage -name \*.so` | grep -C 6 libm4ri-
 0x0000000000000000 (NULL)               0x0

File: /usr/lib64/python2.7/site-packages/sage/modules/vector_mod2_dense.so

Dynamic section at offset 0x15cd0 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x3e20
 0x000000000000000d (FINI)               0x12734
 0x0000000000000019 (INIT_ARRAY)         0x215cb8
--

Dynamic section at offset 0x10b870 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libbrial.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libbrial_groebner.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x21c18
--

File: /usr/lib64/python2.7/site-packages/sage/matrix/matrix_mod2_dense.so

Dynamic section at offset 0x38c60 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgd.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x6ea8
 0x000000000000000d (FINI)               0x32dc0
 0x0000000000000019 (INIT_ARRAY)         0x238c48
--

File: /usr/lib64/python2.7/site-packages/sage/matrix/matrix_gf2e_dense.so

Dynamic section at offset 0x2ec50 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm4rie-0.0.20150908.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x6400
 0x000000000000000d (FINI)               0x28150
 0x0000000000000019 (INIT_ARRAY)         0x22ec38

Only 4 in total. And sage/rings/polynomial/polynomial_gf2x.so is not one of them.

readelf -d /usr/lib64/python2.7/site-packages/sage/rings/polynomial/polynomial_gf2x.so

Dynamic section at offset 0x38c10 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libntl.so.31]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x7510

[fbissey]

Arghhh! sage/rings/polynomial/polynomial_gf2x.pyx uses 2 function from m4ri and both of them are macros, which is why libm4ri doesn't appear in the final link.

[fbissey]

It seems that all the users of m4ri currently have an entry with appropriate flags in module_list.py so removing the line from m4ri.pxd works. But this has to be sorted in a better way.

[fbissey]

We have compile! Documentation building in progress. It may take a little bit of time before I can run doctests but we are at a new stage.

[dimpase]

Replying to fbissey:

OK, was your problem

[sagelib-7.5.beta3] [ 1/59] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p3/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.o -std=c99 -I/Users/fbissey/build/sage/local/include/m4ri -I/Users/fbissey/build/sage/local/include -g -fPIC -Wall -O2 -mmmx -msse -msse2 -msse3 -fno-strict-aliasing
[sagelib-7.5.beta3] error: invalid argument '-std=c99' not allowed with 'C++/ObjC++'

right, this one.

[jdemeyer]

Something like ​https://github.com/cython/cython/pull/466 could help to better organize compiler flags, but upstream Cython doesn't seem to like it.

[dimpase]

With a ​tip by Min, I am also able to complete the Linux/clang (4.0.0) build.

(pyzmq still would need a fix---hopefully upstream...)

[dimpase]

the results of make ptest are

sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # 16 doctests failed
sage -t src/sage/plot/graphics.py  # 1 doctest failed
sage -t src/sage/plot/plot.py  # 1 doctest failed
sage -t src/sage/plot/plot3d/shapes2.py  # 1 doctest failed
sage -t src/sage/tests/cmdline.py  # 2 doctests failed
sage -t src/sage/libs/gap/all_documented_functions.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/libs/gap/assigned_names.py  # Timed out
sage -t src/sage/plot/plot3d/base.pyx  # 4 doctests failed
sage -t src/sage/schemes/elliptic_curves/heegner.py  # 37 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_rational_field.py  # 50 doctests failed
sage -t src/sage/tests/book_stein_ent.py  # 3 doctests failed
sage -t src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t src/sage/plot/plot3d/platonic.py  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/padics.py  # 28 doctests failed
sage -t src/sage/schemes/elliptic_curves/sha_tate.py  # 26 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_generic.py  # 14 doctests failed
sage -t src/sage/schemes/elliptic_curves/BSD.py  # 21 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_point.py  # 13 doctests failed
sage -t src/sage/schemes/elliptic_curves/height.py  # 2 doctests failed
sage -t src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 11 doctests failed
sage -t src/sage/libs/eclib/interface.py  # 34 doctests failed
sage -t src/sage/schemes/elliptic_curves/padic_lseries.py  # 2 doctests failed
sage -t src/sage/libs/eclib/mwrank.pyx  # 3 doctests failed
sage -t src/sage/homology/simplicial_complex.py  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t src/sage/misc/benchmark.py  # 3 doctests failed
sage -t src/sage/tests/benchmark.py  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/ell_tate_curve.py  # 4 doctests failed
sage -t src/sage/libs/lcalc/lcalc_Lfunction.pyx  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/ell_egros.py  # 4 doctests failed
sage -t src/sage/interfaces/mwrank.py  # 5 doctests failed
sage -t src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t src/sage/rings/padics/CR_template.pxi  # 2 doctests failed
sage -t src/sage/schemes/elliptic_curves/constructor.py  # 4 doctests failed
sage -t src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 2376.2 seconds
    cpu time: 6923.9 seconds
    cumulative wall time: 17065.3 seconds

looks like main issues are with number theory things like elliptic curves.

[fbissey]

Well this is mine on OS X

sage -t --long src/sage/algebras/group_algebra.py  # 5 doctests failed
sage -t --long src/sage/algebras/quaternion_algebra_element.py  # 4 doctests failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra.py  # 8 doctests failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra_element.pyx  # 38 doctests failed
sage -t --long src/sage/arith/misc.py  # 20 doctests failed
sage -t --long src/sage/calculus/calculus.py  # 19 doctests failed
sage -t --long src/sage/categories/quotient_fields.py  # 2 doctests failed
sage -t --long src/sage/categories/rings.py  # 8 doctests failed
sage -t --long src/sage/crypto/lattice.py  # 1 doctest failed
sage -t --long src/sage/crypto/mq/sr.py  # 7 doctests failed
sage -t --long src/sage/geometry/polyhedron/palp_database.py  # 1 doctest failed
sage -t --long src/sage/interfaces/singular.py  # 1 doctest failed
sage -t --long src/sage/libs/pynac/pynac.pxd  # 1 doctest failed
sage -t --long src/sage/matrix/matrix2.pyx  # 3 doctests failed
sage -t --long src/sage/matrix/matrix_integer_dense.pyx  # 13 doctests failed
sage -t --long src/sage/matrix/matrix_rational_dense.pyx  # 1 doctest failed
sage -t --long src/sage/misc/functional.py  # 1 doctest failed
sage -t --long src/sage/modular/etaproducts.py  # 10 doctests failed
sage -t --long src/sage/modules/free_module_integer.py  # 57 doctests failed
sage -t --long src/sage/modules/vector_space_morphism.py  # 3 doctests failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/plot/plot3d/base.pyx  # 4 doctests failed
sage -t --long src/sage/plot/plot3d/platonic.py  # 1 doctest failed
sage -t --long src/sage/plot/plot3d/shapes2.py  # 1 doctest failed
sage -t --long src/sage/repl/preparse.py  # 1 doctest failed
sage -t --long src/sage/rings/complex_number.pyx  # 4 doctests failed
sage -t --long src/sage/rings/integer_ring.pyx  # 5 doctests failed
sage -t --long src/sage/rings/misc.py  # 1 doctest failed
sage -t --long src/sage/rings/real_mpfi.pyx  # 3 doctests failed
sage -t --long src/sage/rings/real_mpfr.pyx  # 1 doctest failed
sage -t --long src/sage/rings/ring.pyx  # 1 doctest failed
sage -t --long src/sage/rings/tests.py  # 2 doctests failed
sage -t --long src/sage/rings/finite_rings/residue_field.pyx  # 14 doctests failed
sage -t --long src/sage/rings/number_field/galois_group.py  # 8 doctests failed
sage -t --long src/sage/rings/number_field/number_field_base.pyx  # 15 doctests failed
sage -t --long src/sage/rings/number_field/number_field_rel.py  # 10 doctests failed
sage -t --long src/sage/rings/number_field/order.py  # 3 doctests failed
sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage -t --long src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long src/sage/rings/polynomial/polynomial_element.pyx  # 7 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx  # 6 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_number_field.pyx  # 19 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/constructor.py  # 11 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/ell_egros.py  # 9 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/ell_rational_field.py  # 15 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/heegner.py  # 33 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/schemes/generic/algebraic_scheme.py  # 10 doctests failed
sage -t --long src/sage/schemes/projective/projective_morphism.py  # 16 doctests failed
sage -t --long src/sage/schemes/toric/divisor.py  # 2 doctests failed
sage -t --long src/sage/schemes/toric/ideal.py  # 45 doctests failed
sage -t --long src/sage/schemes/toric/points.py  # 1 doctest failed
sage -t --long src/sage/schemes/toric/variety.py  # 4 doctests failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/structure/element.pyx  # 5 doctests failed
sage -t --long src/sage/structure/formal_sum.py  # 1 doctest failed
sage -t --long src/sage/tests/french_book/calculus_doctest.py  # 3 doctests failed
sage -t --long src/sage/tests/french_book/domaines_doctest.py  # 3 doctests failed
sage -t --long src/sage/tests/french_book/polynomes.py  # 1 doctest failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/doc/en/prep/Quickstarts/Abstract-Algebra.rst  # 3 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 3 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/nf_introduction.rst  # 5 doctests failed

There is one segfault related to brial to investigate. A lot of plotting fail a specific comparison and there is a strange problem with fplll

        from fpylll import LLL, IntegerMatrix
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/__init__.py", line 6, in <module>
        from .fplll.enumeration import Enumeration, EnumerationError
    ImportError: dlopen(/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so, 2): Symbol not found: __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd
      Referenced from: /Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so
      Expected in: flat namespace
     in /Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so

I am guessing the symbol is not in the "flat namespace". That warning happens a lot in plots:

sage -t --long src/sage/plot/plot.py
**********************************************************************
File "src/sage/plot/plot.py", line 1678, in sage.plot.plot.plot
Failed example:
    plot(2*x+1,(x,0,5),ticks=[[0,1,e,pi,sqrt(20)],2],tick_formatter="latex")
Expected:
    Graphics object consisting of 1 graphics primitive
Got:
    doctest:warning
      File "/Users/fbissey/build/sage-clang/src/bin/sage-runtests", line 89, in <module>
        err = DC.run()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/control.py", line 1130, in run
        self.run_doctests()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/control.py", line 854, in run_doctests
        self.dispatcher.dispatch()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1700, in dispatch
        self.parallel_dispatch()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1590, in parallel_dispatch
        w.start()  # This might take some time
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1866, in start
        super(DocTestWorker, self).start()
      File "/Users/fbissey/build/sage-clang/local/lib/python/multiprocessing/process.py", line 130, in start
        self._popen = Popen(self)
      File "/Users/fbissey/build/sage-clang/local/lib/python/multiprocessing/forking.py", line 126, in __init__
        code = process_obj._bootstrap()
      File "/Users/fbissey/build/sage-clang/local/lib/python/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1839, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2132, in __call__
        runner.run(test)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 636, in run
        return self._run(test, compileflags, out)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.plot.plot.plot[99]>", line 1, in <module>
        plot(Integer(2)*x+Integer(1),(x,Integer(0),Integer(5)),ticks=[[Integer(0),Integer(1),e,pi,sqrt(Integer(20))],Integer(2)],tick_formatter="latex")
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 765, in displayhook
        plain_text, rich_output = self._rich_output_formatter(obj, dict())
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 623, in _rich_output_formatter
        rich_output = self._call_rich_repr(obj, rich_repr_kwds)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 583, in _call_rich_repr
        return obj._rich_repr_(self)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/plot/graphics.py", line 908, in _rich_repr_
        self.save, kwds, file_ext, output_container)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 711, in graphics_from_save
        save_function(filename, **kwds)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/misc/decorators.py", line 473, in wrapper
        return func(*args, **kwds)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/plot/graphics.py", line 3206, in save
        figure.tight_layout()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/figure.py", line 1754, in tight_layout
        rect=rect)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/tight_layout.py", line 349, in get_tight_layout_figure
        pad=pad, h_pad=h_pad, w_pad=w_pad)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/tight_layout.py", line 126, in auto_adjust_subplotpars
        tight_bbox_raw = union([ax.get_tightbbox(renderer) for ax in subplots])
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 3679, in get_tightbbox
        bb_xaxis = self.xaxis.get_tightbbox(renderer)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1073, in get_tightbbox
        ticks_to_draw = self._update_ticks(renderer)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1038, in _update_ticks
        if not mtransforms.interval_contains(interval_expanded, loc):
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/transforms.py", line 2780, in interval_contains
        ((a < b) and (a <= val and b >= val))
    :
    RuntimeWarning: invalid value encountered in greater_equal
    Graphics object consisting of 1 graphics primitive

[fbissey]

And the segfaults

• sage/rings/polynomial/multi_polynomial_sequence.py

  sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 1089 ##
  0
  sage: B.<a,b,c,d> = BooleanPolynomialRing() ## line 1145 ##
  sage: F = Sequence([c + d + b + 1, a + c + d, a*b + c, b*c*d + c]) ## line 1146 ##
  sage: F.eliminate_linear_variables() # everything vanishes ## line 1147 ##
  ------------------------------------------------------------------------
  0   signals.so                          0x000000010604db21 print_backtrace + 65
  1   signals.so                          0x000000010605115c sigdie + 60
  2   signals.so                          0x00000001060510cf cysigs_signal_handler + 351
  3   libsystem_platform.dylib            0x00007fffbdabcbba _sigtramp + 26
  4   ???                                 0x0000000105a5ee00 0x0 + 4389727744
  5   pbori.so                            0x000000019198d45f _ZN8polybori8groebner11linalg_stepERNSt3__16vectorINS_15BoolePolynomialENS1_9allocatorIS3_EEEENS_8BooleSetES8_bbPKc + 479
  6   pbori.so                            0x000000019198cf43 _ZN8polybori8groebner14gauss_on_polysERKNSt3__16vectorINS_15BoolePolynomialENS1_9allocatorIS3_EEEE + 275
  7   pbori.so                            0x000000019198cc50 _ZL58__pyx_pw_4sage_5rings_10polynomial_5pbori_39gauss_on_polysP7_objectS0_ + 192
  8   libpython2.7.dylib                  0x000000010591714a PyEval_EvalFrameEx + 18394
  9   libpython2.7.dylib                  0x000000010591a3e2 fast_function + 338
  

• sage/rings/polynomial/pbori.pyx (says abort but segfault a lot before aborting)

  sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 4670 ##
  0
  sage: B.<x0,x1,x2,x3> = BooleanPolynomialRing(4) ## line 4689 ##
  sage: I = B.ideal((x0 + x1 + x2 + x3, x0*x1 + x1*x2 + x0*x3 + x2*x3, x0*x1*x2 + x0*x1*x3 + x0*x2*x3 + x1*x2*x3, x0*x1*x2*x3 + 1)) ## line 4690 ##
  sage: gb = I.groebner_basis() ## line 4694 ##
  sage: f,g,h,i = I.gens() ## line 4695 ##
  sage: f.reduce(gb) ## line 4696 ##
  sage: p = f*g + x0*h + x2*i ## line 4698 ##
  sage: p.reduce(gb) ## line 4699 ##
  sage: p.reduce(I) ## line 4701 ##
  Assertion failed: (result != map.end()), function get, file ../../groebner/include/polybori/groebner/PolyEntryIndices.h, line 115.
  ------------------------------------------------------------------------
  0   signals.so                          0x0000000102c38b21 print_backtrace + 65
  1   signals.so                          0x0000000102c3c15c sigdie + 60
  2   signals.so                          0x0000000102c3c0ad cysigs_signal_handler + 317
  3   libsystem_platform.dylib            0x00007fffbdabcbba _sigtramp + 26
  4   ???                                 0x00007fb02d196af0 0x0 + 140394647612144
  5   libsystem_c.dylib                   0x00007fffbd943420 abort + 129
  6   libsystem_c.dylib                   0x00007fffbd90a893 basename_r + 0
  7   libbrial_groebner.0.dylib           0x000000018e7df737 _ZNK8polybori8groebner16PolyEntryIndices3getINSt3__13mapINS_13BooleExponentEiNS_13MapComparatorIS5_EENS3_9allocatorINS3_4pairIKS5_iEEEEEES5_EEiRKT_RKT0_NS1_7uncheckE + 551
  8   libbrial_groebner.0.dylib           0x000000018e7df4ff _ZNK8polybori8groebner16PolyEntryIndices3getINS1_7uncheckEEEiRKNS_13BooleExponentET_ + 47
  9   libbrial_groebner.0.dylib           0x000000018e7df4bd _ZNK8polybori8groebner16PolyEntryIndicesclINS_13BooleExponentEEEiRKT_ + 29
  10  libbrial_groebner.0.dylib           0x000000018e79805c _ZNK8polybori8groebner15PolyEntryVector5indexINS_13BooleExponentEEEmRKT_ + 44
  11  libbrial_groebner.0.dylib           0x000000018e7e79ad _ZN8polybori8groebner15PolyEntryVectorclINS_13BooleExponentEEENS0_18PolyEntryReferenceERKT_ + 61
  12  libbrial_groebner.0.dylib           0x000000018e819ea7 _ZNK8polybori8groebner20SetAssociatedMinimalINS_13BooleExponentELb0EEclERKS2_ + 39
  13  libbrial_groebner.0.dylib           0x000000018e81808a _ZN8polybori8groebner17ReductionStrategy28unmarkNonMinimalLeadingTermsENS_8BooleSetE + 186
  14  libbrial_groebner.0.dylib           0x000000018e817e16 _ZN8polybori8groebner17ReductionStrategy19setupSetsForElementERKNS0_9PolyEntryE + 262
  15  pbori.so                            0x000000018e5410c1 _ZL76__pyx_pw_4sage_5rings_10polynomial_5pbori_17ReductionStrategy_5add_generatorP7_objectS0_ + 177
  16  pbori.so                            0x000000018e4cdc9a _ZL25__Pyx_PyObject_CallOneArgP7_objectS0_ + 154
  17  pbori.so                            0x000000018e529621 _ZL71__pyx_pw_4sage_5rings_10polynomial_5pbori_17BooleanPolynomial_131reduceP7_objectS0_ + 2289
  18  libpython2.7.dylib                  0x000000010250614a PyEval_EvalFrameEx + 18394
  

[fbissey]

Replying to jdemeyer:

Something like ​https://github.com/cython/cython/pull/466 could help to better organize compiler flags, but upstream Cython doesn't seem to like it.

Yes it would help. I am a bit unwilling to ship a change of that kind if upstream is unwilling. They don't seem to be interested. In the meantime what should we do with that line in pynac.pxd? If we don't have that functionality in cython it will have to go for now.

I also had found a way to get -std=c99 included in CFLAGS provided by m4ri.pc upstream but after that experience, I am not so sure I want it added there anymore. That's asking for trouble when you mix languages. You'd literally need a break down in CPPFLAGS, CFLAGS, CXXFLAGS and so on, to avoid accidents.

[dimpase]

I've installed libc++, and while rebuilding from scratch using it, hit a bug in linbox: (in linbox/field/gf2.h)

#if !defined(__PATHCC__) && !(defined(__APPLE__) && defined(__clang__))
#define stdBitReference std::_Bit_reference
#else
#define stdBitReference std::vector<bool>::reference
#endif

that appears to assume that clang not on OSX much be using libstdc++.

By the way, I am still using low-level gcc ABI, e.g. this is a typical ldd output on a C++ library (notice libgcc_s.so there):

$ ldd local/lib/libgmpxx.so
	linux-vdso.so.1 (0x00007ffc1a59f000)
	libgmp.so.16 => /home/dima/Sage/sage-tree/local/lib/libgmp.so.16 (0x00007f5ef4323000)
	libc++.so.1 => /usr/lib64/libc++.so.1 (0x00007f5ef403c000)
	libcxxrt.so.1 => /usr/lib64/libcxxrt.so.1 (0x00007f5ef3e18000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libgcc_s.so.1 (0x00007f5ef3c01000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f5ef3904000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f5ef356a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5ef334e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f5ef3149000)
	/lib64/ld-linux-x86-64.so.2 (0x00005610d175f000)

[fbissey]

Interesting. I cannot comment on why clang doesn't provide its own libgcc_s or equivalent. Hum, output of readelf -d local/lib/libgmpxx.so and ldd -r /usr/lib64/libc++.so.1 please!

As for linbox I think this is ridiculous. libstdc++ and libc++ are supposed to be feature complete to C++11. Surely you can use the same declaration with both. Issue to open with linbox in my opinion.

[dimpase]

Replying to fbissey:

Interesting. I cannot comment on why clang doesn't provide its own libgcc_s or equivalent. Hum, output of readelf -d local/lib/libgmpxx.so and ldd -r /usr/lib64/libc++.so.1 please!

What is the replacement of libgcc_s? compiler-rt or libcxxabi? It is probably due to me using clang-3.9 rather than clang-9999 that it uses libgcc_s.

As for linbox I think this is ridiculous. libstdc++ and libc++ are supposed to be feature complete to C++11. Surely you can use the same declaration with both. Issue to open with linbox in my opinion.

OK, I've opened #21977.

[dimpase]

output of readelf on libgmpxx

[dimpase]

output of readelf on libc++

[dimpase]

the next problem is in building sagelib, and there things are totally misconfigured: matrix_modn_dense_float.cpp is compiled with clang, not with clang++ (and with option -std=gnu++11 for some reason) I get an error saying, most probably, that it does not expect C++ here...

[sagelib-7.5.beta3] [  1/237] clang -fno-strict-aliasing -OPT:Olimit=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/dima/Sage/sage-tree/local/lib/python2.7/site-packages/cysignals -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local/include/python2.7 -I/home/dima/Sage/sage-tree/local/lib/python2.7/site-packages/numpy/core/include -I/home/dima/Sage/sage-tree/src -I/home/dima/Sage/sage-tree/src/sage/ext -I/home/dima/Sage/sage-tree/src/build/cythonized -I/home/dima/Sage/sage-tree/src/build/cythonized/sage/ext -I/home/dima/Sage/sage-tree/local/include/python2.7 -c /home/dima/Sage/sage-tree/src/build/cythonized/sage/matrix/matrix_modn_dense_float.cpp -o build/temp.linux-x86_64-2.7/home/dima/Sage/sage-tree/src/build/cythonized/sage/matrix/matrix_modn_dense_float.o -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -fPIC -std=gnu++11 -msse4.1 -mfma -mavx2 -DFFLAS_COMPILED -DFFPACK_COMPILED -fPIC -std=gnu++11 -I/home/dima/Sage/sage-tree/local/include/linbox -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local//include -I/home/dima/Sage/sage-tree/local/include -fPIC -std=gnu++11 -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local//include -I/home/dima/Sage/sage-tree/local/include -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -fPIC -std=gnu++11 -msse4.1 -mfma -mavx2 -DFFLAS_COMPILED -DFFPACK_COMPILED -fPIC -std=gnu++11 -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local//include -I/home/dima/Sage/sage-tree/local/include -fno-strict-aliasing

I wonder why the clang build of sagelib works on OSX - is it because distutils understand what to do in this case?

[fbissey]

I occasionally have seen .cpp files compiled with clang usually without trouble. And matrix_modn_dense_float is one such case. What is the error exactly?

[jdemeyer]

Replying to dimpase:

the next problem is in building sagelib, and there things are totally misconfigured: matrix_modn_dense_float.cpp is compiled with clang, not with clang++ (and with option -std=gnu++11 for some reason)

This is ​https://bugs.python.org/issue1222585 (an 11-year old bug!)

[fbissey]

I am familiar with it. You wouldn't believe how broken building python or python package is broken on AIX and that's one of the referred bugs for that. On AIX I have to hand hold the building of matplotlib, numpy and scipy, compiling manually the stuff that fails because of wrong compilers and sometimes wrong compiler flags. The way it is broken feels completely magical.

[dimpase]

Replying to jdemeyer:

Replying to dimpase:

the next problem is in building sagelib, and there things are totally misconfigured: matrix_modn_dense_float.cpp is compiled with clang, not with clang++ (and with option -std=gnu++11 for some reason)

This is ​https://bugs.python.org/issue1222585 (an 11-year old bug!)

they say it is enhancement. Technically speaking, building C++ extensions is just not supported... We should just switch to Julia :-)

Anyhow, I suppose the problem I face is that clang has no idea about STL library implementation it should use (well, clang++ does have an appropriate flag (-stdlib=), clang probably just ignores such a flag). It's just an accident that it works with gcc.

[fbissey]

Can you just post the error, so we can see it?

[jdemeyer]

Replying to dimpase:

It's just an accident that it works with gcc.

Not really because gcc looks at the extension of the file to be compiled and decides what to do depending on that. If you compile a .cpp file with gcc, I guess it behaves exactly the same as compiling that file with g++. Doesn't clang do the same?

[dimpase]

Replying to fbissey:

Can you just post the error, so we can see it?

 clang -fno-strict-aliasing -OPT:Olim
it=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/hom
e/dima/Sage/sage-tree/local/lib/python2.7/site-packages/cysignals 
-I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-t
ree/local/include/python2.7 -I/home/dima/Sage/sage-tree/local/lib/
python2.7/site-packages/numpy/core/include -I/home/dima/Sage/sage-
tree/src -I/home/dima/Sage/sage-tree/src/sage/ext -I/home/dima/Sag
e/sage-tree/src/build/cythonized -I/home/dima/Sage/sage-tree/src/b
uild/cythonized/sage/ext -I/home/dima/Sage/sage-tree/local/include
/python2.7 -c /home/dima/Sage/sage-tree/src/build/cythonized/sage/
matroids/lean_matrix.c -o build/temp.linux-x86_64-2.7/home/dima/Sa
ge/sage-tree/src/build/cythonized/sage/matroids/lean_matrix.o -fno
-strict-aliasing
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/src/build/cythonized/sage/matrix/matrix_modn_dense_float.cpp:564
:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/matrix/dense-matrix.h:79:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/matrix/densematrix/blas-matrix.h:42:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/vector/stream.h:872:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/vector/stream-gf2.h:77:
[sagelib-7.5.beta3] /home/dima/Sage/sage-tree/local/include/linbox
/field/gf2.h:1008:10: fatal error: '__bit_reference' file not foun
d
[sagelib-7.5.beta3] #include <__bit_reference>

__bit_reference is a header file from libc++; which clang++ finds as I invoke it as clang++ -stdlib=libc++. But clang has no idea about this parameter, and thus it cannot find it. It is probably not simply including the corresponding directory with -I parameter, as more things from STL are most probably have to be included by default.

[dimpase]

Replying to jdemeyer:

Replying to dimpase:

It's just an accident that it works with gcc.

Not really because gcc looks at the extension of the file to be compiled and decides what to do depending on that. If you compile a .cpp file with gcc, I guess it behaves exactly the same as compiling that file with g++. Doesn't clang do the same?

I am not sure. By the way, the file in question, where the error pops up, (local/include/linbox/fields/gf2.h) is a C++ file, but is has extension .h, indicating plain C file, I think. (well, it is included from a .cpp file, so in that sense it is meant to be C++).

Besides, g++ has parameters that gcc does not have, such as -nostdinc++ (needed if you want use an STL implementation different from libstdc++), and so I don't see how this could at all fly with gcc if these g++-only parmeters are to be invoked.

[fbissey]

Well, gcc and clang are not the real compilers, they are drivers. In the case of gcc the real compiler is cc1 for example. So you can pass a c++ file to compile to gcc, and it may be recognised as c++ and passed to cc1plus but the flags that are usually set by g++ won't be present, only the one set by gcc. In the case of gnu compilers it looks like gcc and g++ have the same include flags.

So for example, while it is possible to produce a .o with gcc from a C++ source (and same with clang if you are lucky) it is impossible to link a C++ program with gcc/clang instead of g++/clang++ (without extra manual flags that is) because only the C++ driver knows to add the right flags for the standard C++ library.

I notice for example that matrix_modn_dense_float.cpp is compiled with clang on OS X but the .so file is linked with clang++.

[fbissey]

On OS X, all the .cpp files are compiled with clang and then linked with clang++. No exceptions. I cannot check the equivalent on linux until tomorrow but I suspect it is the same.

[fbissey]

The version of python shipped in Gentoo actually includes ​​https://bugs.python.org/issue1222585 (and yes that means in sage-on-gentoo g++ is used for compiling in the right places). I would be supportive to include that patch in sage's python for once.

[dimpase]

Replying to fbissey:

The version of python shipped in Gentoo actually includes ​​https://bugs.python.org/issue1222585 (and yes that means in sage-on-gentoo g++ is used for compiling in the right places). I would be supportive to include that patch in sage's python for once.

How does one find out what gentoo is including here?

[fbissey]

Replying to dimpase:

Replying to fbissey:

The version of python shipped in Gentoo actually includes ​​https://bugs.python.org/issue1222585 (and yes that means in sage-on-gentoo g++ is used for compiling in the right places). I would be supportive to include that patch in sage's python for once.

How does one find out what gentoo is including here?

I am assuming you want to brew your own :) In your DISTFILES folder (by default /usr/portage/distfiles) there should be a python-gentoo-patches-2.7.10-0.tar.xz tarball (and a 2.7.12-0 one if you upgraded recently) there are a few other things but inside that tarball are all main - unconditional - patches. patches/21_all_distutils_c++.patch inside that tarball is what you are after if you want to test it. It should be identical to what's on the python tracker for python 2.7 since it's the same author....

[fbissey]

I really need another machine than my OS X laptop to work on this. Do we have some resources I could tap remotely?

[dimpase]

Replying to fbissey:

I really need another machine than my OS X laptop to work on this. Do we have some resources I could tap remotely?

Not sure if anyone has such resources in the open; indeed, it's illegal to run OSX on non-Apple hardware, although the net is full of instructions on how to e.g. hack VMWarePlayer to make it run OSX on non-Apple hardware.

I'm thinking that perhaps resurrecting FreeBSD port, which as of version 10 has clang as the default toolchain, is the best bet for a platform on which port to clang can be completed without extra hassles.

[fbissey]

OS X, sage 7.6.rc2

----------------------------------------------------------------------
sage -t --long src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage -t --long src/sage/plot/plot.py  # Timed out after testing finished
sage -t --long src/sage/schemes/projective/projective_morphism.py  # 16 doctests failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/ell_rational_field.py  # 17 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/heegner.py  # 33 doctests failed
sage -t --long src/sage/arith/misc.py  # 20 doctests failed
sage -t --long src/sage/matrix/matrix2.pyx  # 3 doctests failed
sage -t --long src/sage/rings/tests.py  # 2 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_element.pyx  # 7 doctests failed
sage -t --long src/sage/libs/ppl.pyx  # Timed out after testing finished
sage -t --long src/sage/misc/functional.py  # 1 doctest failed
sage -t --long src/sage/structure/element.pyx  # 5 doctests failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/sage/matrix/matrix_integer_dense.pyx  # 13 doctests failed
sage -t --long src/sage/schemes/generic/algebraic_scheme.py  # 10 doctests failed
sage -t --long src/sage/schemes/toric/divisor.py  # 2 doctests failed
sage -t --long src/sage/schemes/toric/variety.py  # 4 doctests failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra.py  # 8 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 3 doctests failed
sage -t --long src/sage/calculus/calculus.py  # 19 doctests failed
sage -t --long src/sage/tests/french_book/calculus_doctest.py  # 3 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/crypto/mq/sr.py  # 7 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/constructor.py  # 11 doctests failed
sage -t --long src/sage/rings/real_mpfi.pyx  # 3 doctests failed
sage -t --long src/sage/rings/complex_number.pyx  # 4 doctests failed
sage -t --long src/sage/schemes/toric/points.py  # 1 doctest failed
sage -t --long src/sage/libs/pynac/pynac.pxd  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/ell_egros.py  # 9 doctests failed
sage -t --long src/sage/rings/real_mpfr.pyx  # 1 doctest failed
sage -t --long src/sage/rings/number_field/number_field_rel.py  # 10 doctests failed
sage -t --long src/sage/rings/finite_rings/residue_field.pyx  # 14 doctests failed
sage -t --long src/sage/rings/number_field/order.py  # 3 doctests failed
sage -t --long src/sage/rings/number_field/galois_group.py  # 8 doctests failed
sage -t --long src/sage/modular/etaproducts.py  # 10 doctests failed
sage -t --long src/sage/tests/french_book/polynomes.py  # 1 doctest failed
sage -t --long src/sage/tests/french_book/domaines_doctest.py  # 3 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/nf_introduction.rst  # 5 doctests failed
sage -t --long src/sage/algebras/group_algebra.py  # 5 doctests failed
sage -t --long src/sage/rings/ring.pyx  # 1 doctest failed
sage -t --long src/doc/en/prep/Quickstarts/Abstract-Algebra.rst  # 3 doctests failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/modules/vector_space_morphism.py  # 3 doctests failed
sage -t --long src/sage/modules/free_module_integer.py  # 57 doctests failed
sage -t --long src/sage/geometry/polyhedron/palp_database.py  # 1 doctest failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra_element.pyx  # 38 doctests failed
sage -t --long src/sage/rings/integer_ring.pyx  # 5 doctests failed
sage -t --long src/sage/categories/rings.py  # 8 doctests failed
sage -t --long src/sage/categories/quotient_fields.py  # 2 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx  # 6 doctests failed
sage -t --long src/sage/matrix/matrix_rational_dense.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/toric/ideal.py  # 45 doctests failed
sage -t --long src/sage/structure/formal_sum.py  # 1 doctest failed
sage -t --long src/sage/repl/preparse.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_number_field.pyx  # 19 doctests failed
sage -t --long src/sage/rings/number_field/number_field_base.pyx  # 15 doctests failed
sage -t --long src/sage/crypto/lattice.py  # 1 doctest failed
sage -t --long src/sage/rings/misc.py  # 1 doctest failed
sage -t --long src/sage/algebras/quaternion_algebra_element.py  # 4 doctests failed
----------------------------------------------------------------------

[fbissey]

OK so diving down. There are two main errors that spams a lot of doctests. One is in polybori - go figure :( - and the other is in fplll. So for fplll we have

    ImportError: dlopen(/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so, 2): Symbol not found: __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd
      Referenced from: /Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so

Searching inside libfplll.dylib we get the following information

nm local/lib/libfplll.2.dylib | grep EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd
000000000000cc50 t __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd

The leading lowercase "t" means that the symbol is "local". Which is probably why enumeration.so cannot see it. That's clearly a subtle problem in the compilation of fplll.

[dimpase]

A lot of these are related to elliptic curves, more precisely, to eclib interface with sage. This seems to be the 3rd group of errors.

I'll see the author of fplll in few days here in Oxford, perhaps we can have a look.

[fbissey]

@ isuruf you wouldn't have something for the polybori issue would you?

[fbissey]

With the patch from isuruf at #22643 we are way down in doctests failures

sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage -t --long src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/sage/crypto/mq/sr.py  # 7 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/libs/pynac/pynac.pxd  # 1 doctest failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed

[isuruf]

Replying to fbissey:

@ isuruf you wouldn't have something for the polybori issue would you?

nope. I have access to osx only through travis-ci. I'm waiting for ​https://github.com/cython/cython/pull/466 to be merged.

[fbissey]

Replying to isuruf:

Replying to fbissey:

@ isuruf you wouldn't have something for the polybori issue would you?

nope. I have access to osx only through travis-ci. I'm waiting for ​https://github.com/cython/cython/pull/466 to be merged.

OK, I had to ask. Now that fplll is sorted, things are clearer. We have

• polybori segfaulting,
• missing symbols in pynac - I will get in touch upstream,
• issues with matplotlib, although it could be interaction with sage types.
• similarly an issue in numpy.
• a tolerance problem in lseries_ell.py, the doctest look at a zero value within 1e-17, but we are only 2e-16 of it.

[dimpase]

Let's revive the FreeBSD port. FreeBSD as of version 10 uses clang by default, and is closer to OSX in other ways too, so it would be a good free testing ground for the use of clang with Sage. Preliminary checks show that it's relatively little work. I've created #22679 for this.

[fbissey]

Using BRiAl git master fix the the segfault, forgeting pynac which is in his own ticket, so we are down to

sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed

matplotlib issue:

...
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1038, in _update_ticks
        if not mtransforms.interval_contains(interval_expanded, loc):
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/transforms.py", line 2780, in interval_contains
        ((a < b) and (a <= val and b >= val))
    :
    RuntimeWarning: invalid value encountered in greater_equal

for all of them.

sage -t --long --warn-long 62.6 src/sage/schemes/elliptic_curves/lseries_ell.py
**********************************************************************
File "src/sage/schemes/elliptic_curves/lseries_ell.py", line 377, in sage.schemes.elliptic_curves.lseries_ell.Lseries_ell.twist_values
Failed example:
    vals  # abs tol 1e-17
Expected:
    [(-11, 1.47824342), (-8, 8.9590946e-18), (-7, 1.85307619), (-4, 2.45138938)]
Got:
    [(-11, 1.47824342), (-8, -1.59574955e-16), (-7, 1.85307619), (-4, 2.45138938)]
Tolerance exceeded in 1 of 8:
    8.9590946e-18 vs -1.59574955e-16, tolerance 2e-16 > 1e-17
**********************************************************************

And

sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed

numpy issue

        numpy.float32('1.5') * x
    :
    RuntimeWarning: invalid value encountered in multiply

both of them.

[fbissey]

Turns out the pynac issue is not a proper issue. cython was trying to run the test with g++ instead of clang++ and the mixing was not working unsurprisingly.

[fbissey]

I now have a working environment where I can do the work on linux and hope it will match the problems on OS X. Have to figure how not build gcc though, as clang passes itself as gcc 4.2.1 which triggers toolchain building and my sage-env hack is OS X specific.

Inserting a clang identification logic isn't easy. It can be done. Interestingly autoconf people would probably say we are wrong to select versions, we should look for compiler features. Admittedly, that's more difficult than just banning gcc versions, even if it is the right thing.

[dimpase]

Replying to fbissey:

I now have a working environment where I can do the work on linux and hope it will match the problems on OS X.

For clang++, do you use libc++, as OSX does, rather than libstdc++, as a non-bleeding-edge clang-on-Linux would use the latter... (having clang++/libc++ on Linux looked like a very experimental thing to me; without libc++ some C++-specific thing are rather different)

Have to figure how not build gcc though, as clang passes itself as gcc 4.2.1 which triggers toolchain building and my sage-env hack is OS X specific.

how about SAGE_INSTALL_GCC=no? Is this not enough?

Inserting a clang identification logic isn't easy. It can be done. Interestingly autoconf people would probably say we are wrong to select versions, we should look for compiler features. Admittedly, that's more difficult than just banning gcc versions, even if it is the right thing.

[fbissey]

I have to figure it out but gentoo stabilised clang-3.9 and you can set it up to use libcxx, which I did. Hope there is no extra flags involved.

Checking on SAGE_INSTALL_GCC=no, didn't really think about that. It should work in that particular case. But I already started to put some logic in configure.ac to figure out we are using clang. On OS X where I didn't have a fortran compiler to start with, it would result in an error - of course because I don't have a fortran compiler I want to build gcc. Just not use gcc and g++.

[fbissey]

Now I have a full logic that automatically skips toolchain building if using clang. Forbidding older version of clang needs another macro (I already added one) but is doable.

diff --git a/configure.ac b/configure.ac
index 52a9d2c..fa3d1d8 100644
--- a/configure.ac
+++ b/configure.ac
@@ -442,20 +442,30 @@ elif test x$GFC != xyes; then
 else
     # Since need_to_install_gcc is "no", we know that
     # at least C, C++ and Fortran compilers are available.
-    # We also know that all compilers are GCC.
-
+    # We also know that all compilers are GCC-like.
     # Find out the compiler versions:
     AX_GCC_VERSION()
     AX_GXX_VERSION()
-
-    # Add the .0 because Debian/Ubuntu gives version numbers like
-    # 4.6 instead of 4.6.4 (Trac #18885)
-    case "$GXX_VERSION.0" in
-        [[0-3]].*|4.[[0-7]].*)
-            # Install our own GCC if the system-provided one is older than gcc-4.8.
-            SAGE_SHOULD_INSTALL_GCC([you have $CXX version $GXX_VERSION, which is quite old]);;
-    esac
-
+    
+    #Figuring out if we are using clang instead of gcc.
+    AC_LANG_PUSH(C) # we are still set on FORTRAN
+    AX_COMPILER_VENDOR()
+    AC_LANG_POP([C])
+    IS_CLANG=no
+    if test "x$ax_cv_c_compiler_vendor" = xclang
+    then IS_CLANG=yes
+    fi
+    if test "x$IS_CLANG" = "xno"
+    then
+        # for gcc we know we want more recent versions
+        # Add the .0 because Debian/Ubuntu gives version numbers like
+        # 4.6 instead of 4.6.4 (Trac #18885)
+        case "$GXX_VERSION.0" in
+            [[0-3]].*|4.[[0-7]].*)
+                # Install our own GCC if the system-provided one is older than gcc-4.8.
+                SAGE_SHOULD_INSTALL_GCC([you have $CXX version $GXX_VERSION, which is quite old]);;
+        esac
+    fi
     # The following tests check that the version of the compilers
     # are all the same.
     if test "$GCC_VERSION" != "$GXX_VERSION"; then
@@ -497,7 +507,10 @@ else
 
     if test "x$fortran_version_string" = x
     then
-        SAGE_SHOULD_INSTALL_GCC([the Fortran compiler is not the same version as the C compiler])
+        if test "x$IS_CLANG" = "xno"
+        then
+           SAGE_SHOULD_INSTALL_GCC([the Fortran compiler is not the same version as the C compiler])
+        fi
     fi
 fi

and using ​http://www.gnu.org/software/autoconf-archive/ax_compiler_vendor.html

It assumes that you are using the same vendor for CC and CXX. Mixing those would probably trigger an error in configure unless the gnu compiler you are trying to mix with clang is version 4.2.1 precisely.

On OS X where there is no fortran compiler by default, gcc would still be build and the sage-env hack necessary. But you should be able to bring an external gfortran and skip the building of the toolchain.

[jdemeyer]

You can simplify test "x$IS_CLANG" = "xno" to test "$IS_CLANG" = no or even test $IS_CLANG = no if you know that $IS_CLANG is one word :-)

I think it would be more logical to test AX_COMPILER_VENDOR() earlier, when you are actually testing the C++ compiler.

For Fortran, maybe just drop the version check altogether? If clang works with gfortran, then I would guess that gcc/g++ also work with different versions of gfortran.

[fbissey]

Replying to jdemeyer:

You can simplify test "x$IS_CLANG" = "xno" to test "$IS_CLANG" = no :-)

Yes, it went through several iterations before I figured out that I was trying to check the vendor of the fortran compiler :( [which is unknown because the macro only knows how to do C and C++ compilers].

I think it would be more logical to test AX_COMPILER_VENDOR() earlier, when you are actually testing the C++ compiler.

Yes it would make sense.

For Fortran, maybe just drop the version check altogether? If clang works with gfortran, then I would guess that gcc/g++ also work with different versions of gfortran.

Most likely. I would imagine that if it doesn't it is actually broken.

I am not doing a branch at this stage, I am just gathering material for other people to be able to do it themselves (and give me some feedback like you just did of course).

[fbissey]

And now for something ridiculous. libgap fails to build with clang-3.9.1 on linux

libtool: link: clang -shared   .libs/libgap_la-ariths.o .libs/libgap_la-c_random.o .libs/libgap_la-gmpints.o .libs/libgap_la-objccoll.o .libs/libgap_la-rational.o .libs/libgap_la-system.o .libs/libgap_la-blister.o .libs/libgap_la-c_type1.o .libs/libgap_la-gvars.o .libs/libgap_la-objcftl.o .libs/libgap_la-read.o .libs/libgap_la-tietze.o .libs/libgap_la-bool.o .libs/libgap_la-cyclotom.o .libs/libgap_la-integer.o .libs/libgap_la-objects.o .libs/libgap_la-records.o .libs/libgap_la-vars.o .libs/libgap_la-calls.o .libs/libgap_la-dt.o .libs/libgap_la-intfuncs.o .libs/libgap_la-objfgelm.o .libs/libgap_la-saveload.o .libs/libgap_la-vec8bit.o .libs/libgap_la-c_filt1.o .libs/libgap_la-dteval.o .libs/libgap_la-intrprtr.o .libs/libgap_la-objpcgel.o .libs/libgap_la-scanner.o .libs/libgap_la-vecffe.o .libs/libgap_la-c_meths1.o .libs/libgap_la-exprs.o .libs/libgap_la-iostream.o .libs/libgap_la-objscoll.o .libs/libgap_la-sctable.o .libs/libgap_la-vecgf2.o .libs/libgap_la-code.o .libs/libgap_la-finfield.o .libs/libgap_la-libgap.o .libs/libgap_la-opers.o .libs/libgap_la-set.o .libs/libgap_la-vector.o .libs/libgap_la-compiler.o .libs/libgap_la-funcs.o .libs/libgap_la-listfunc.o .libs/libgap_la-permutat.o .libs/libgap_la-stats.o .libs/libgap_la-weakptr.o .libs/libgap_la-compstat.o .libs/libgap_la-gap.o .libs/libgap_la-listoper.o .libs/libgap_la-plist.o .libs/libgap_la-streams.o .libs/libgap_la-c_oper1.o .libs/libgap_la-lists.o .libs/libgap_la-precord.o .libs/libgap_la-string.o .libs/libgap_la-costab.o .libs/libgap_la-gasman.o .libs/libgap_la-macfloat.o .libs/libgap_la-range.o .libs/libgap_la-sysfiles.o .libs/libgap_la-pperm.o .libs/libgap_la-trans.o .libs/libgap_la-profile.o    -L/home/fbissey/sandbox/git-fork/sage-clang/local/lib -lgmp -lm  -Wl,--version-script=../src/libgap.map -Wl,-rpath -Wl,/home/fbissey/sandbox/git-fork/sage-clang/local/lib   -Wl,-soname -Wl,libgap.so.4 -o .libs/libgap.so.4.8.6
/usr/bin/x86_64-pc-linux-gnu-ld: .libs/libgap_la-ariths.o: relocation R_X86_64_32S against `libGAP_ZEROOp' can not be used when making a shared object; recompile with -fPIC
.libs/libgap_la-ariths.o: error adding symbols: Bad value
clang-3.9: error: linker command failed with exit code 1 (use -v to see invocation)

As it turns out this bit

CPPFLAGS="$CPPFLAGS"' -DSYS_DEFAULT_PATHS=\"'"$SAGE_LOCAL/gap/latest"'\"'

about SYS_DEFAULT_PATHS is breaking the detection of -fPIC -DPIC by autoconf, so everything is compiled without -fPIC and the linking of the shared object fails.

At this point I think

1. if it is useful make it a configuration option
2. I don't any of that in sage-on-gentoo. Is this really necessary? I guess I'll figure the answer to that shortly.

[fbissey]

Oh, I see only used for the testsuite. May be I can move that logic there then.

[dimpase]

yes, I have seen the need for missing -fPIC with clang too, only I never knew why this was broken...

[fbissey]

Hum, yes. Adding the linbox annoyance #21977 to the list. At least that's easily patchable.

[fbissey]

Well pyzmq fails to build.

    clang -fno-strict-aliasing -OPT:Olimit=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/fbissey/sandbox/git-fork/sage-clang/local/include -Izmq/utils -Izmq/backend/cython -Izmq/devices -c build/temp.linux-x86_64-2.7/scratch/vers.c -o build/temp.linux-x86_64-2.7/scratch/vers.o
    clang build/temp.linux-x86_64-2.7/scratch/vers.o -L/home/fbissey/sandbox/git-fork/sage-clang/local/lib -R/home/fbissey/sandbox/git-fork/sage-clang/local/lib -lzmq -lrt -o build/temp.linux-x86_64-2.7/scratch/vers
    Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory


    error: Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory

clang doesn't know to pass -R directly to the linker. For gcc distutils uses -Wl,-R but recons other compilers don't need to do that. From python2.7/distutils/unixccompiler.py

    def runtime_library_dir_option(self, dir):
        # XXX Hackish, at the very least.  See Python bug #445902:
        # http://sourceforge.net/tracker/index.php
        #   ?func=detail&aid=445902&group_id=5470&atid=105470
        # Linkers on different platforms need different options to
        # specify that directories need to be added to the list of
        # directories searched for dependencies when a dynamic library
        # is sought.  GCC has to be told to pass the -R option through
        # to the linker, whereas other compilers just know this.
        # Other compilers may need something slightly different.  At
        # this time, there's no way to determine this information from
        # the configuration data stored in the Python installation, so
        # we use this hack.
        compiler = os.path.basename(sysconfig.get_config_var("CC"))
        if sys.platform[:6] == "darwin":
            # MacOSX's linker doesn't understand the -R flag at all
            return "-L" + dir
        elif sys.platform[:7] == "freebsd":
            return "-Wl,-rpath=" + dir
        elif sys.platform[:5] == "hp-ux":
            if self._is_gcc(compiler):
                return ["-Wl,+s", "-L" + dir]
            return ["+s", "-L" + dir]
        elif sys.platform[:7] == "irix646" or sys.platform[:6] == "osf1V5":
            return ["-rpath", dir]
        elif self._is_gcc(compiler):
            return "-Wl,-R" + dir
        else:
            return "-R" + dir

_is_gcc basically returns true if gcc or g++ is part of the filename of the compiler. So clang on linux, gets the last option when it would need the same as gcc, or freeBSD. And I would also like people who wrote that code to read ld man page so they drop the way they use -R. It is very much a legacy behavior.

[fbissey]

Started doctesting, definitely need the new eclib. I get all those backtraces right now.

[dimpase]

Replying to fbissey:

Started doctesting, definitely need the new eclib. I get all those backtraces right now.

there was a return missing in a function computing double, and clang does not forgive this Matlab-speak

[fbissey]

Replying to fbissey:

Well pyzmq fails to build.

    clang -fno-strict-aliasing -OPT:Olimit=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/fbissey/sandbox/git-fork/sage-clang/local/include -Izmq/utils -Izmq/backend/cython -Izmq/devices -c build/temp.linux-x86_64-2.7/scratch/vers.c -o build/temp.linux-x86_64-2.7/scratch/vers.o
    clang build/temp.linux-x86_64-2.7/scratch/vers.o -L/home/fbissey/sandbox/git-fork/sage-clang/local/lib -R/home/fbissey/sandbox/git-fork/sage-clang/local/lib -lzmq -lrt -o build/temp.linux-x86_64-2.7/scratch/vers
    Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory


    error: Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory

clang doesn't know to pass -R directly to the linker. For gcc distutils uses -Wl,-R but recons other compilers don't need to do that. From python2.7/distutils/unixccompiler.py

    def runtime_library_dir_option(self, dir):
        # XXX Hackish, at the very least.  See Python bug #445902:
        # http://sourceforge.net/tracker/index.php
        #   ?func=detail&aid=445902&group_id=5470&atid=105470
        # Linkers on different platforms need different options to
        # specify that directories need to be added to the list of
        # directories searched for dependencies when a dynamic library
        # is sought.  GCC has to be told to pass the -R option through
        # to the linker, whereas other compilers just know this.
        # Other compilers may need something slightly different.  At
        # this time, there's no way to determine this information from
        # the configuration data stored in the Python installation, so
        # we use this hack.
        compiler = os.path.basename(sysconfig.get_config_var("CC"))
        if sys.platform[:6] == "darwin":
            # MacOSX's linker doesn't understand the -R flag at all
            return "-L" + dir
        elif sys.platform[:7] == "freebsd":
            return "-Wl,-rpath=" + dir
        elif sys.platform[:5] == "hp-ux":
            if self._is_gcc(compiler):
                return ["-Wl,+s", "-L" + dir]
            return ["+s", "-L" + dir]
        elif sys.platform[:7] == "irix646" or sys.platform[:6] == "osf1V5":
            return ["-rpath", dir]
        elif self._is_gcc(compiler):
            return "-Wl,-R" + dir
        else:
            return "-R" + dir

_is_gcc basically returns true if gcc or g++ is part of the filename of the compiler. So clang on linux, gets the last option when it would need the same as gcc, or freeBSD. And I would also like people who wrote that code to read ld man page so they drop the way they use -R. It is very much a legacy behavior.

upstream issue: ​http://bugs.python.org/issue25229

[dimpase]

Replying to fbissey:

Oh, I see only used for the testsuite. May be I can move that logic there then.

I presume you mean CPPFLAGS from comment 150. Do you have a ticket for this?

[fbissey]

Not yet. I haven't fixed the testsuite so far, but I guess that's not an obstacle to opening a new ticket.

[dimpase]

Replying to fbissey:

Not yet. I haven't fixed the testsuite so far, but I guess that's not an obstacle to opening a new ticket.

This is now #22784

[fbissey]

I know I said I wouldn't provide a branch, but all in all we are closer to clang support out of the box than I thought we were last week.

So it is more important that things to start the build are available and open to criticism.

---

New commits:

​28bea10	Basic changes to get building of gcc to skip if using clang+gfortran from system and always use clang on OS X
​376ed4f	Add missing macro

[dimpase]

can we have the following

+    if [ "$UNAME" = "Darwin" ]; then
+        CXX=clang++

less Darwin-centric? (I'd like to have "Darwin" || "FreeBSD")

Perhaps CC should come from a configure parameter (that has a default)

Then, how about making CPP configurable too? (I imagine that for clang on linux you don't want the default cpp, no?)

[fbissey]

I would have thought that you would bring your fortran compiler on freeBSD. That part is only really activated if you compile gcc, which is only necessary if you don't already have a gfortran install.

Anything more complicated should indeed be the object of a configuration.

For CPP, on linux I have clang-cpp installed, yes there is room for stuff there, especially after the experience with the curl ticket.

[jdemeyer]

where we will assume there were only installed to provide gfortran

Are you sure that it's impossible to just install gfortran without gcc and g++?

That would be the best solution: don't install gcc and g++ at all if clang is available. Then you would need less special cases in sage-env.

[fbissey]

I don't think it is possible to build gfortran without building gcc/g++, however gfortran is not bootstrapped it is built with the c or c++ compiler so something may be possible. Also there is the possibility of only installing the fortran bits - provided no other dependencies are needed. I'll look into it.

[dimpase]

Replying to fbissey:

I would have thought that you would bring your fortran compiler on freeBSD. That part is only really activated if you compile gcc, which is only necessary if you don't already have a gfortran install.

I am not sure if system's gfortran is a good idea on FreeBSD, especially as it needs that ugly /lib/libgcc_s replacement. Perhaps Sage's gfortran will not need it. I'd like to try, as replacing /lib/libgcc_s might be a red flag for many...

By the way, what is the correct way to set up global LDFLAGS for Sage building? (Adding them by hand is ugly...)

[fbissey]

Ok first investigations in gfortran

1. easiest: install the binary package from ​https://gcc.gnu.org/wiki/GFortranBinaries#MacOS
2. I tried to follow ​http://fortranhelp.blogspot.co.nz/2010/09/building-gfortran-from-source-code-on.html but it sets itself to build the c compiler as well.

Yes libgcc_c is probably why I end up build the c compiler.

[fbissey]

Number 2 still installed gcc but not g++. It was also much, much faster.

[dimpase]

Replying to fbissey:

Yes libgcc_c is probably why I end up build the c compiler.

do you mean libgcc_s ?

[fbissey]

Replying to dimpase:

Replying to fbissey:

Yes libgcc_c is probably why I end up build the c compiler.

do you mean libgcc_s ?

yes

[fbissey]

OK after a good night sleep and some thinking I think we can tweak the configuration script and the gcc package to trigger the building of gfortan only under certain condition. At install time we can just suppress the gcc and cpp executables.

It is also much faster. It takes several hours (~3) to bootstrap and build a full gcc/g++/gfortran on my poor laptop. It is now the the single most time consuming item, probably longer than the rest combined. It would be reduced to about half an hour which makes building on that machine much more bearable.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​74cb5f7

Only build gfortran if clang is CC and gfortran is not found.

[fbissey]

OK this is a first draft. It is quite generic in that if CC is clang and gfortran is not found the gcc package does a minimal install of gfortran.

In the current setup there is no way to force a full gcc build if you wished too. In fact I removed some OS X specific bits for now until I have checked they are still necessary (I don't believe so but I need experimental proof). So right now the branch would fail to build a full gcc on OS X anyway.

I have leveraged sage-env-config in a minimal way, I am open to alternative suggestion as I put something there that is only ever used by the gcc spkg, which makes it an irritant.

Adding spkg-install.in for gcc could be an interesting option and make it more controllable by configure.

[fbissey]

Argh.... mpir gets initially build without C++ support when building the toolchain. And I have disabled the rebuilding if you just install gfortran. Back to the drawing board.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2ce0c32

Restore unconditional re-building of gmp/mpir/mpfr/mpc

[fbissey]

Going back one step. That little adventure made me think we need to leverage autoconf more at the spkg-install level but that could be done orthogonally to this ticket.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fca8460

Improve the comments in gcc

[fbissey]

OK this is fairly good. With this ticket and #22677 sage builds on OS X with clang out of the box and install just gfortran (and a few other bits of gcc along the way). And only the following doctests are failing

sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed

one instance of numerical noise (lseries_ell.py) and the rest is unexpected warning messages from numpy. Needs to be investigated but everything is functional.

[dimpase]

on freebsd/clang numpy produces warnings like

        numpy.float32('1.5') * x
    :
    RuntimeWarning: invalid value encountered in multiply
    1.50000000000000*x

(this one from src/sage/plot/plot.py) Do you get the same?

The biggest difference is that on freebsd I also get problems with sympow, it does not fully work on some curves...

[fbissey]

Yes, same. sage/structure/coerce.pyx same, sage/rings/polynomial/polynomial_real_mpfr_dense.pyx same. The other ones are

RuntimeWarning: invalid value encountered in greater_equal

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d472cca

Merge branch 'develop' into clang-start

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eab0de5

generate new configure tarball

[jhpalmieri]

It is exciting to see this progress, and it is exciting to have the gcc build take only 7 minutes on my Mac, since it only has to build fortran.

[jhpalmieri]

Do we have any good benchmarks? I don't know of any reason why Sage built this way should differ in performance compared to Sage built with gcc, but we should probably test it.

[fbissey]

Replying to jhpalmieri:

Do we have any good benchmarks? I don't know of any reason why Sage built this way should differ in performance compared to Sage built with gcc, but we should probably test it.

I don't know that we have. But moving to clang on OS X just makes sense. We had to include numerous work around for gcc itself, and some package are already compiled with clang on a case by case basis. This is because we will continue having problems with gcc until either

• gcc adopt the same extensions as clang
• apple decides it isn't a good thing to use clang extension in their libc headers and fixes it

Neither is happening anytime soon as far as I am aware. It is increasingly difficult to build a big project on OS X with gcc/g++. The general mood is to just bite the bullet and move to clang.

[jhpalmieri]

Don't misunderstand me, I am happy to see the change. We should make sure it doesn't lead to any regressions, though.

[fbissey]

OK, benchmarking has always been a thorny issue. I don't believe we have one apart from doctesting time.

[jhpalmieri]

I tried some ad hoc testing, and I don't see any speed difference between the two builds. I don't know if anyone wants to try anything systematic.

[dimpase]

Replying to jhpalmieri:

I tried some ad hoc testing, and I don't see any speed difference between the two builds. I don't know if anyone wants to try anything systematic.

Internet has benchmarking stuff for clang vs gcc, in particular as far as HPC things are concerned. Give or take, they are largely on par. (One nice thing about clang is that it compiles and links faster than gcc, this is something seemingly well-known).

By the way, what is the assembler used by clang on OSX? Is it built-in, or external?

[jhpalmieri]

Replying to dimpase:

By the way, what is the assembler used by clang on OSX? Is it built-in, or external?

I can check. How do I identify the assembler?

[jdemeyer]

I seem to recall that clang has a built-in assembler. But don't quote me on this...

[jhpalmieri]

The man page says:

Clang also supports the use of an integrated assembler

and lists as options

-integrated-as, -no-integrated-as
       Used to enable and disable, respectively,  the  use  of  the  integrated
       assembler.  Whether  the integrated assembler is on by default is target
       dependent.

[fbissey]

Interesting

Mirage:~ fbissey$ clang -cc1 --help | grep assembler
  -emit-llvm              Use the LLVM representation for assembler and object files
  -fno-integrated-as      Disable the integrated assembler
  -massembler-fatal-warnings
                          Make assembler warnings fatal

So an option to disabler internal assembler is lister but not one to enable it, which suggest that it is the default.

[jhpalmieri]

I'm building Sage with SAGE_CHECK=yes on OS X, and I am seeing some failures of test suites. For some packages (ntl, gf2x) I see failures when building with gcc or with clang, but with zeromq, I see a new failure just with clang:

Invalid argument (stream_engine.cpp:94)
/bin/sh: line 1: 58001 Abort trap: 6           ${dir}$tst
FAIL: test_shutdown_stress

[fbissey]

Yes, I haven't done that extensively with my little laptop. I will investigate zeromq - I think it is due a version bump anyway, our pyzmq complains that zeromq is too old. Version bump could take care of the failure, or not.

[fbissey]

OK, very easy solution to that.

[zeromq-4.0.5] Assertion failed: (rc == 0), function main, file test_abstract_ipc.cpp, line 31.
[zeromq-4.0.5] /bin/sh: line 1:  8898 Abort trap: 6           ${dir}$tst
[zeromq-4.0.5] XFAIL: test_abstract_ipc
[zeromq-4.0.5] PASS: test_many_sockets
[zeromq-4.0.5] PASS: test_shutdown_stress
[zeromq-4.0.5] PASS: test_pair_ipc
[zeromq-4.0.5] PASS: test_reqrep_ipc
[zeromq-4.0.5] PASS: test_timeo
[zeromq-4.0.5] PASS: test_fork
[zeromq-4.0.5] =====================================================
[zeromq-4.0.5] All 43 tests behaved as expected (1 expected failure)
[zeromq-4.0.5] =====================================================
[zeromq-4.0.5] Making check in tools
[zeromq-4.0.5] make[3]: Nothing to be done for `check'.
[zeromq-4.0.5] make[3]: Nothing to be done for `check-am'.
[zeromq-4.0.5] 
[zeromq-4.0.5] real	0m8.181s
[zeromq-4.0.5] user	0m0.985s
[zeromq-4.0.5] sys	0m1.320s
[zeromq-4.0.5] Deleting temporary build directory
[zeromq-4.0.5] /Users/fbissey/build/sage-clang/local/var/tmp/sage/build/zeromq-4.0.5
[zeromq-4.0.5] Finished installing zeromq-4.0.5.spkg

The gentoo ebuild for zeromq notes that upstream doesn't support testing in parallel. Just doing

diff --git a/build/pkgs/zeromq/spkg-check b/build/pkgs/zeromq/spkg-check
index 598280def1..3b36b91b55 100755
--- a/build/pkgs/zeromq/spkg-check
+++ b/build/pkgs/zeromq/spkg-check
@@ -6,5 +6,5 @@ if [ "$UNAME" = "Darwin" ]; then
 fi
 
 cd src
-$MAKE check
+$MAKE -j1 check
 

makes the test pass for me.

[fbissey]

I have looked into ntl and gf2x. I am assuming that it is in fact the same, or related, failure in both case. I think we should report it to upstream gf2x.

[jhpalmieri]

Is it the same as #18882?

[fbissey]

Replying to jhpalmieri:

Is it the same as #18882?

Yes.

[gf2x-1.1.p1] -n 100x100 
[gf2x-1.1.p1] failed : '100 100 cca56aa6 76540123' != '100 100 2269c4fa 76540123'

[jhpalmieri]

ccache also fails for me, both gcc and clang builds. I just opened #22836.

[jhpalmieri]

Another clang-specific failure on OS X: giac. On one hand, giac builds in 35sec with clang, as compared to 6m with gcc (!), but I get one test suite failure with clang:

51c51
< -sqrt(6)*I*sqrt(sqrt(13)+3),sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
---
> sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
FAIL: chk_fhan16

[jhpalmieri]

Cython fails its test suite, with both clang and gcc, reported at #22166.

[jhpalmieri]

Note that if a package fails its test suite with both gcc and clang, I am not at all saying that we need to deal with it before this ticket gets positively reviewed. I'm just reporting the current situation. When clang introduces new failures, we should investigate.

[jhpalmieri]

Replying to fbissey:

OK, very easy solution to that.

[ snip ]

The gentoo ebuild for zeromq notes that upstream doesn't support testing in parallel. Just doing

diff --git a/build/pkgs/zeromq/spkg-check b/build/pkgs/zeromq/spkg-check
index 598280def1..3b36b91b55 100755
--- a/build/pkgs/zeromq/spkg-check
+++ b/build/pkgs/zeromq/spkg-check
@@ -6,5 +6,5 @@ if [ "$UNAME" = "Darwin" ]; then
 fi
 
 cd src
-$MAKE check
+$MAKE -j1 check
 

makes the test pass for me.

The test passed for me without any changes on a different machine, so it is intermittent. We should probably make this change anyway. See #22837. (I've made you the author and me the reviewer.)

[fbissey]

Replying to jhpalmieri:

Another clang-specific failure on OS X: giac. On one hand, giac builds in 35sec with clang, as compared to 6m with gcc (!), but I get one test suite failure with clang:

51c51
< -sqrt(6)*I*sqrt(sqrt(13)+3),sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
---
> sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
FAIL: chk_fhan16

Most interesting because it originates with ​https://github.com/sagemath/sage/blob/master/build/pkgs/giac/patches/cSolveorder-check.patch I am sure. And that patch is needed with gcc for pari-2.8+ (i.e. with pari 2.7 it is not needed).

[jhpalmieri]

cysignals also fails its test suite. This is not fixed by #22695: that is, both the current version and the version in #22695 pass tests with gcc and fail with clang. The failures:

**********************************************************************
File "src/cysignals/tests.pyx", line 168, in tests.pyx
Failed example:
    test_sig_on()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 182, in tests.pyx
Failed example:
    test_sig_str()
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: Everything ok!
Got:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest tests.pyx[12]>", line 1, in <module>
        test_sig_str()
      File "src/cysignals/tests.pyx", line 189, in cysignals.tests.test_sig_str (build/src/cysignals/tests.c:2665)
        sig_str("Everything ok!")
      File "src/cysignals/signals.pyx", line 109, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1699)
        raise SignalError(s or "Segmentation fault")
    SignalError: Everything ok!
**********************************************************************
File "src/cysignals/tests.pyx", line 203, in tests.pyx
Failed example:
    test_sig_on_cython()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 220, in tests.pyx
Failed example:
    test_sig_on_cython_except()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 238, in tests.pyx
Failed example:
    test_sig_on_cython_except_all()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 267, in tests.pyx
Failed example:
    test_sig_check_inside_sig_on()
Expected:
    KeyboardInterrupt()
Got:
    RuntimeError('Aborted',)
Traceback (most recent call last):
  File "rundoctests.py", line 20, in <module>
    failures, _ = doctest.testfile(f, module_relative=False, optionflags=flags)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 2036, in testfile
    runner.run(test)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 1454, in run
    return self.__run(test, compileflags, out)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs
  File "src/cysignals/signals.pyx", line 252, in cysignals.signals.python_check_interrupt (build/src/cysignals/signals.c:2858)
    sig_check()
  File "src/cysignals/signals.pyx", line 97, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1307)
    raise KeyboardInterrupt
KeyboardInterrupt
make[3]: *** [check-doctest] Interrupt: 2

[jhpalmieri]

eclib fails its test suite with clang but not with gcc. Looks like just numerical noise, I hope:

Testing smattest...
Testing comptest...
5c5
< AGM((3.125,4.25),(1,0)) = (2.07831617217326684,1.5633615899065958201)
---
> AGM((3.125,4.25),(1,0)) = (2.07831617217326684,1.56336158990659582)
14c14
< whose cube is    (2.9999999999999999998,3.9999999999999999999)
---
> whose cube is    (2.9999999999999999999,4)
16c16
< whose cube is    (2.9999999999999999997,3.9999999999999999999)
---
> whose cube is    (2.9999999999999999996,3.9999999999999999998)
make[4]: *** [check_procs] Error 1
make[3]: *** [check] Error 2
make[2]: *** [check-recursive] Error 1
Error: eclib's test suite failed to pass.

[fbissey]

Hum, cysignals stuff looks serious, looks like some mapping is wrong. For eclib it really looks like just numerical noise.

[fbissey]

I can reproduce the eclib failures on linux+clang.

[jhpalmieri]

That's my complete list of failed test suites. Summarizing: clang-only failures for:

• eclib: numerical noise, so hopefully minor
• giac
• cysignals

Failures with both clang and gcc:

• ccache: #22836
• gf2x and ntl (related?)
• zeromq: intermittent, perhaps addressed at #22837 although I haven't tested thoroughly to see if that completely fixes the problem
• cython: #22166

Oh, and I forgot to mention: python3 – does any python package ever pass its test suite with Sage?

[fbissey]

Replying to jhpalmieri:

Oh, and I forgot to mention: python3 – does any python package ever pass its test suite with Sage?

I don't think we have in some time but Volker would have some ideas on that.

[fbissey]

I just finished a build of 8.0.beta2 with clang+linux, hacking my way through the pyzmq problem, and there is a truckload of doctest failures. However it is exclusively, as far as I can see through the mess, all done to a failure in sage.matrix.matrix_integer_dense.so. A nice isolated example is

sage -t --long src/sage/rings/qqbar.py
**********************************************************************
File "src/sage/rings/qqbar.py", line 237, in sage.rings.qqbar
Failed example:
    r.imag().minpoly()  # long time (10s on sage.math, 2013)
Exception raised:
    Traceback (most recent call last):
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 503, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 866, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.qqbar[72]>", line 1, in <module>
        r.imag().minpoly()  # long time (10s on sage.math, 2013)
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/rings/qqbar.py", line 3403, in minpoly
        self._minimal_polynomial = self._descr.minpoly()
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/rings/qqbar.py", line 6426, in minpoly
        self._minpoly = self._value.minpoly()
      File "sage/rings/number_field/number_field_element.pyx", line 4291, in sage.rings.number_field.number_field_element.NumberFieldElement_absolute.minpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/rings/number_field/number_field_element.cpp:40412)
        return self.charpoly(var, algorithm).radical() # square free part of charpoly
      File "sage/rings/number_field/number_field_element.pyx", line 4258, in sage.rings.number_field.number_field_element.NumberFieldElement_absolute.charpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/rings/number_field/number_field_element.cpp:40187)
        return R(self.matrix().charpoly())
      File "sage/matrix/matrix_rational_dense.pyx", line 945, in sage.matrix.matrix_rational_dense.Matrix_rational_dense.charpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_rational_dense.c:11214)
        f = A.charpoly(var, algorithm='linbox')
      File "sage/matrix/matrix_integer_dense.pyx", line 1251, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.charpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_integer_dense.c:13209)
        g = self._charpoly_linbox(var)
      File "sage/matrix/matrix_integer_dense.pyx", line 1304, in sage.matrix.matrix_integer_dense.Matrix_integer_dense._charpoly_linbox (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_integer_dense.c:14045)
        return self._poly_linbox(var=var, typ='charpoly')
      File "sage/matrix/matrix_integer_dense.pyx", line 1327, in sage.matrix.matrix_integer_dense.Matrix_integer_dense._poly_linbox (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_integer_dense.c:14435)
        sig_on()
      File "src/cysignals/signals.pyx", line 109, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1699)
    SignalError: Segmentation fault
**********************************************************************
1 item had failures:

[jdemeyer]

Replying to jhpalmieri:

That's my complete list of failed test suites. Summarizing: clang-only failures for:

• cysignals

I just had some "fun" debugging this. Turns out it is a 9-year old(!) LLVM bug: ​https://bugs.llvm.org/show_bug.cgi?id=965

There is a work-around in #22691.

[jhpalmieri]

Replying to jdemeyer:

Replying to jhpalmieri:

That's my complete list of failed test suites. Summarizing: clang-only failures for:

• cysignals

I just had some "fun" debugging this. Turns out it is a 9-year old(!) LLVM bug: ​https://bugs.llvm.org/show_bug.cgi?id=965

There is a work-around in #22691.

Great! That works for me on OS X.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ec06d97	Merge branch 'develop' into clang-start
​caa9ac9	bumping configure tarball

[rws]

Replying to dimpase:

The biggest difference is that on freebsd I also get problems with sympow, it does not fully work on some curves...

This may be the same as what I get on OpenSuSE, invalid pointers in sympow leading to runtime errors and failing to compute the analytic rank. I opened #22862.

[jhpalmieri]

Following up on a comment from #22756: right now we build python2, python3 (as of #22756), curl, psutil, and R using clang rather than gcc, because of troubles with gcc.

With this branch, if a user on OS X happens to have installed gcc (for example, version 5.4.0 in /usr/local/bin), then those packages will break if you use that version of gcc to compile them. So if here or on a followup ticket, we are hoping to simplify the logic in some of the spkg-install scripts, we have to be careful. We either need to keep the parts of the script that say

if [ "$UNAME" = "Darwin" ] && [ $MACOSX_VERSION -ge 16 ]; then
    echo "OS X 10.$[$MACOSX_VERSION-4] Building with clang."
    CC=clang
fi

or we need to force the user to compile everything with clang even if they have a good version of gcc installed.

[fbissey]

Well, I think I will want to force clang. At the moment we keep adding package exceptions to use clang - and I mean clang but not clang++. This is important, the libc is provided by the system for both gcc and clang, so apart from a few isolated exceptions you can mix c compilers.

Now if we happen to want to make a clang++ exception for a package, and it is not stand alone - ie. it provides a library that will be consumed by sage or other packages, then we will have no choice but convert all the dependency chain to clang++. This is because clang++ and g++ each have their own runtimes and you cannot mix them up (#22798). Technically they should be ABI compatible (provided the proper incantation), the reality is that it is madness.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2dec934	Merge branch 'develop' into clang-start
​678988c	bump configure tarball

[dimpase]

isn't it important to explicitly use CC=clang even on OSX? Note that gcc and g++ in Xcode are meant to try to be gcc-compatible:

$ g++ -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Volumes/Sage/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

note --with-gxx-include-dir, absent in cc configuration.

[fbissey]

I don't think so. Yes the gcc/g++ compiler drivers gives a bit more info than cc or clang

Mirage:~ fbissey$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Mirage:~ fbissey$ cc -v
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Mirage:~ fbissey$ clang -v
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

but I don't think it is that important. Whether you set CC=lang or not, autotools still thinks clang is gcc. We have to use an extra non default macro to figure out it is not gcc. I don't think there is a difference in output whether I set it or not.

[dimpase]

I did set CC and CXX, and on OSX get

[gcc-5.4.0] checking for x86_64-apple-darwin16.5.0-gcc... /Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/xgcc -B/Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/ -B/Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/bin/ -B/Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/lib/ -isystem /Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/include -isystem /Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/sys-include   
[gcc-5.4.0] checking for suffix of object files... configure: error: in `/Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/x86_64-apple-darwin16.5.0/libgcc':
[gcc-5.4.0] configure: error: cannot compute suffix of object files: cannot compile
[gcc-5.4.0] See `config.log' for more details.
[gcc-5.4.0] make[4]: *** [configure-target-libgcc] Error 1

[dimpase]

OSX gcc spkg failure

[fbissey]

Interesting. I don't need the log, I have seen that error so many times in many different context. Basically the gcc you built in the previous stage is not "working", usually because it cannot find gmp/mpfr/mpc or find the right one because of {DY,}LD_LIBRARY_PATH settings or sometimes lack thereof. It should be manifest in the output of

otool -L /Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/xgcc

[fbissey]

Very interesting... Setting CC=clang CXX=clang++ works no problems. Setting CC=cc CXX=c++ leads to the above error and problem in finding libz is the culprit this time.

/Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/cc1:
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/Users/fbissey/build/sage-build/local/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
	/Users/fbissey/build/sage-build/local/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.5.0)
	/Users/fbissey/build/sage-build/local/lib/libgmp.16.dylib (compatibility version 23.0.0, current version 23.2.0)
	libz.so.1.2.8 (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.51.1)

and from the right config.log

configure:3464: /Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/xgcc -B/Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/ -B/Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/bin/ -B/Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/lib/ -isystem /Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/include -isystem /Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/sys-include    -o conftest -g -O2   conftest.c  >&5
dyld: Library not loaded: libz.so.1.2.8
  Referenced from: /Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/cc1
  Reason: image not found
xgcc: internal compiler error: Abort trap: 6 (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.

Of course the very curious bit here is that libz.so has been built instead of libz.dylib and it doesn't have the proper install_name set which leads to the problem. So the real problem is that zlib is built funny when the compiler is set to cc. I expect it doesn't use autotools....

[fbissey]

zlib has a hacked configure script and we get the following results

Mirage:zlib-1.2.8 fbissey$ CC=cc ./configure
Checking for shared library support...
Building shared library libz.so.1.2.8 with cc.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Mirage:zlib-1.2.8 fbissey$ CC=clang ./configure
Checking for shared library support...
Building shared library libz.1.2.8.dylib with clang.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.

So there is detection gap between compiler name and platform somewhere.

[fbissey]

Yes the configure script for zlib is broken in that particular case

case "$cc" in
  *gcc*) gcc=1 ;;
  *clang*) gcc=1 ;;
esac
case `$cc -v 2>&1` in
  *gcc*) gcc=1 ;;
esac

Earlier we have gcc=0, so in theory when CC is either gcc or clang the variable gcc is set to 1, and you are sent on a branch where darwin is identified and the right linking flags are set. However if you set CC=cc the first case statement doesn't do anything, and then the second doesn't either since

Mirage:zlib-1.2.8 fbissey$ cc -v
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

doesn't mention gcc anywhere.

[fbissey]

Upstream fixed the configuration problem (same way I did for experimental purpose), upgrading to 1.2.11 is recommended in another ticket if you think it is essential.

[dimpase]

Replying to fbissey:

Upstream fixed the configuration problem (same way I did for experimental purpose), upgrading to 1.2.11 is recommended in another ticket if you think it is essential.

the update is on #22917 - let's get it done to avoid possible confusion.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​12cf61e

Merge branch 'develop' into clang-start

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cec66df	Merge branch 'develop' into clang-start
​66973af	Paul's workaround from #22799
​b3d23f9	Merge branch 'develop' into clangtrouble
​811ba22	Make the work around only compile with clang
​67fa2fa	bump version
​c881b5d	Merge branch 'clangtrouble' into clang-start
​38f9352	increment configure tarball

[fbissey]

A little word about future plans to finish this now that #22799 seems to have a solution.

• It is now possible to build, and doctest sage itself on OS X with clang building only gfortran with this ticket.
• Similarly on FreeBSD.
• To build with clang on linux we need to patch python (both 2.7 and 3.x) for ​http://bugs.python.org/issue25229 for which I should make a ticket.
• Further on linux which usually default to gcc we want something like #22646, otherwise you have to run sage like CC=clang CXX=clang++ ./sage if you want to avoid problems like the one described at #22678.
• Some packages do not pass their test suite with clang.

After much thinking, while I think the default on OS X should be to use clang, I don't think we should preclude building with gcc, especially if people bring their own gcc. So far the only problems we have seen are in C code. Not C++ and after reflexion I don't think it is going to change. So building with gcc/g++ with clang exception for key packages should work. But apart from those, I think the support for gcc on OS X should become minimal over time.

You could even imagine doing some compile with gcc/clang++/gfortran or clang/g++/gfortran so long as we fix python use of the C compiler to compile C++ code. ​https://bugs.python.org/issue1222585

So the plan forward:

• Fix package testsuite as much as possible - sometimes it will be hard. eclib's tests need to learn about tolerance for example.
• Base future work on #22646
• Make the building of gfortran only, more generic. That is, not relying on a variable telling you the compiler is clang. For that matter we can try to extend this to the case where the platform already has a gcc/g++ compiler but only lacks a fortran compiler. But in a general manner not exposing to the rest of the system that you are not using gcc by a variable like IS_CLANG.

[fbissey]

Replying to fbissey:

I have looked into ntl and gf2x. I am assuming that it is in fact the same, or related, failure in both case. I think we should report it to upstream gf2x.

Confirmed that ntl testsuite failure is due to gf2x. Once I solve the failure of gf2x by not running make tune-toom (​https://trac.sagemath.org/ticket/18882#comment:5) the ntl testsuite complete successfully.

[dimpase]

would you go with having compilers are parameters to configure, or rather stay with env. variables? (IMHO autotools do not offer good support for choosing compilers...)

[fbissey]

Well autotools finds a compiler that works. You can over-ride that by parameters. Note that #22646 makes that choice of compiler stick to runtime unless sage builds its own.

Autotools is system based on testing and finding capabilities of what you have rather than sticking to a brand. Which is what you should wish for. That being said if you don't set CC, AC_PROG_C will go through a default list (which you can override). On OS X, with the current branch it finds Apple's gcc which is really clang and everything is fine.

As far as I am concerned doing CC=.. ./configure.. is not very different to doing something like ./configure --with-c-compiler=....

But we could be enforcing things more at the AC_PROG_{C,CXX,FC} point and even do it by OS. On linux only look for gcc and clang in that order, on OS X and freebsd look for clang and gcc in that order. That's bit harsh but can still be discarded in the event you set CC (I think).

[mkoeppe]

Replying to dimpase:

would you go with having compilers are parameters to configure, or rather stay with env. variables? (IMHO autotools do not offer good support for choosing compilers...)

I'd strongly suggest to stick with using environment variables (note that it is best to pass them to configure as arguments (./configure CC=...), simply because that's the standard way to select a compiler with autotools packages. Best not to invent new sage-specific ways.

[jhpalmieri]

Whatever happens, we need to be careful not to build certain packages on OS X using gcc: in my experience, with Sage's gcc or my own, some packages (like python and R) fail, but they succeed using clang. So whether we use autotools or configure options or something else, we might have to force clang for some packages if we don't force it in general.

I guess we could let the user shoot themselves in the foot by explicitly setting CC etc., but the default settings should work whether or not there is a recent version of gcc in /usr/local/bin, for example.

[fbissey]

Replying to jhpalmieri:

Whatever happens, we need to be careful not to build certain packages on OS X using gcc: in my experience, with Sage's gcc or my own, some packages (like python and R) fail, but they succeed using clang. So whether we use autotools or configure options or something else, we might have to force clang for some packages if we don't force it in general.

Yes, I think we'll still make sure the packages that needs clang on OS X gets it enforced at the spkg-install level.

I guess we could let the user shoot themselves in the foot by explicitly setting CC etc., but the default settings should work whether or not there is a recent version of gcc in /usr/local/bin, for example.

Yes. By default it should take a well supported configuration for the platform. But users should be free to shoot themselves in the foot - best way to learn things, really.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e4384af	Merge branch 'develop' into clang-start
​40d907a	bump configure tarball

[jhpalmieri]

Is there anything I can do to help move this ticket along? What needs to be done now?

[fbissey]

Sorry, I am a bit bogged down in stuff and that has shifted to a lower priority than say, python3. So getting #23046 reviewed would be interesting because you can then test clang on a linux machine. Another ticket in need of attention is #22646 which will need some rebasing for the configure tarball. #22646 is important because it help your compiler to "stick" if you use a compiler than is not in the PATH or not the expected default compiler. Because it touches some of the same files as this ticket, I would very much want it included first.

The next step is to rework the logic so we can trigger the build of gfortran without the building of gcc - as in the attached branch - but without using IS_CLANG. I would like it to be more generic if just gfortran is missing and that includes the case of clang on OS X by default. Bonus point if we can get out of rebuilding mpir, mpfr and mpc after gcc when you build just gfortran.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1b2d84c

Merge branch 'develop' into clang-start

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fb0ff15

new configure tarball

[fbissey]

I think the solution to this will have to be a gfortran package as part of all-toolchain that can only be installed when gcc isn't.

[fbissey]

Say, what is the oldest OS X we support?

[fbissey]

New commits:

​483106a	First draft of moving compiler settings to sage-env-config
​d6e87c8	Merge branch 'develop' into compiler
​6048dff	Forgot to commit the new configure tarball
​f7eff64	Merge branch 'develop' into compiler
​945e824	bump config tarball
​32a0b00	one AC_SUBST per variable only :(
​11dc29a	configure/automake doesn't like variables only defined in conditionals
​ef9f1b3	missing "fi" in sage-env-config.in
​8d8b174	Add a new gfortran spkg and use it when gfortran is not found and you don't have to build the rest of gcc

[fbissey]

OK, revamped branch. This new branch focuses on being able to build gfortran when not found and allow CC/CXX to be clang. It now fully depends on #22646.

[fbissey]

And the current branch is not yet working on OS X. Figuring out why.....

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​40faa0f

updating GFC check section for gfortran only potential install

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bd8c1af

new configure checksums

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3287180

Do not install a C compiler or cpp with the gfortran spkg

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​23c32a4	Small clean up and refining
​77107d8	update config tarball checksum

[fbissey]

OK, people interested should give this branch some trashing - both at the code comment level and testing level.

[jhpalmieri]

When the gfortran package is built on OS X, I also end up with these in local/bin: gcc-ar, gcc-nm, gcc-ranlib, x86_64-apple-darwin16.6.0-gcc, etc. Is that a problem?

[fbissey]

Replying to jhpalmieri:

When the gfortran package is built on OS X, I also end up with these in local/bin: gcc-ar, gcc-nm, gcc-ranlib, x86_64-apple-darwin16.6.0-gcc, etc. Is that a problem?

It shouldn't be. I left the stuff that had a prefix. I only removed the non-bootstraped gcc and cpp binaries that are installed. I believe the rest should be ok to be left alone. I should inspect the gfortran binary package produced upstream. We could be a bit cleaner for all I know.

[fbissey]

I should say a bit more I guess. You ended up with that stuff too with the old branch. While I only want gfortran the build system cannot build it without compiling gcc first - at least I haven't found a way to eliminate that step. I am not even sure that's currently desirable. So a gcc is produced to build gfortran and it is installed.

If you know a way to only install gfortran (even if it still builds a gcc), I am all ears. I will have a look in case it is easy to spot.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a6db502

make some files in the gfortran pkg symlinks to the files in the gcc pkg so that that versions and dependencies are changed at the same time.

[fbissey]

Couldn't find a way to install gfortran separately in gcc's makefiles. In the meantime I made some files in the gfortran folder link to the gcc folder. gcc version bumps are now in sync with changes to gfortran. One would still have to check spkg-install but it makes things easier.

In some ways the work here got a little bit of scope creep. Kind of of a mix of unintended consequences and some thoughts on trying to be generic.:

• As tested by Vincent Delecroix, this ticket allows the version of gfortran to be decoupled from the version of gcc used. It is of course non sense to check the version of the C/C++ compiler matches the fortran one when they may not be from the same vendor. However it could have been kept for gcc specifically. Earlier versions of the branch did.
• I switched from positively checking for clang to checking "is this really gcc?". clang of course pretends to be gcc but so does icc I think. The branch in its current state would allow one to use a compiler that pretends to be gcc to AC_PROG_CC but isn't. This really needs #23046 to work on linux, probably not just for clang.

The ticket keeps the requirement for gfortran but I think that could be relaxed in the future. The real hard requirement is for R. Some R packages don't properly autotool their fortran which means they rely on fortran mangling to be "function_". Any fortran compiler that default to that or can easily be made to do that will do.

[jhpalmieri]

Here is a report about building on OS X:

• if I have my own gfortran present (but no gcc), it builds and passes tests. It does not build gfortran.
• if I have neither gfortran nor gcc, it builds and passes tests. It builds gfortran, of course.
• if I have my own gcc (version 6.3.0), I got an error building libfplll:

  	/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -DFPLLL_DEFAULT_STRATEGY_PATH=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies/default.json\" -I./.. -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -I/include  -fPIC -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/lib -O3 -MT pruner.lo -MD -MP -MF $depbase.Tpo -c -o pruner.lo pruner.cpp &&\
  	mv -f $depbase.Tpo $depbase.Plo
  libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DFPLLL_DEFAULT_STRATEGY_PATH=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies/default.json\" -I./.. -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -I/include -fPIC -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/lib -O3 -MT pruner.lo -MD -MP -MF .deps/pruner.Tpo -c pruner.cpp  -fno-common -DPIC -o .libs/pruner.o
  pruner.cpp: In function 'FT fplll::Pruner<FT>::single_enum_cost(const evec&, std::vector<double>*) [with FT = fplll::FP_NR<long double>]':
  pruner.cpp:247:11: internal compiler error: Segmentation fault: 11
   inline FT Pruner<FT>::single_enum_cost(/*i*/ const evec &b, vector<double> *detailed_cost)
             ^~~~~~~~~~
  
  pruner.cpp:247:11: internal compiler error: Abort trap: 6
  g++: internal compiler error: Abort trap: 6 (program cc1plus)
  Please submit a full bug report,
  with preprocessed source if appropriate.
  See <https://github.com/Homebrew/homebrew-core/issues> for instructions.
  make[6]: *** [pruner.lo] Error 1
  make[6]: *** Waiting for unfinished jobs....
  

  Maybe my version of gcc is broken. Any other ideas of what could be wrong?

[fbissey]

Looks suspiciously like some recent changes to fplll. Let me check...

[fbissey]

OK not what I was thinking. ICE are bad, the question is whether it is something from homebrew or upstream gcc. Does homebrew happen to have fplll in one of their repos? It would be interesting to know if a homebrew version exhibits the same problem.

Alternatively I/you could also try with gcc/gfortran from ​http://hpc.sourceforge.net/

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​777a393

updating configure tarball

[fbissey]

I am trying gcc-7.1 from hpc.sourceforge.net. libfplll compiled with it. I am using gcc/g++/gfortran from that tarball. Checking with someone I know I am surprised you are on gcc-6.3.0 with homebrew. They have 7.1.0 now. Are you working on the polymake package (which apparently doesn't like gcc-7.1)?

[jhpalmieri]

I built and installed gcc 5.4.0 and got errors building libfplll and givaro:

ld: symbol(s) not found for architecture x86_64

Maybe I'm building gcc wrong?

I'm not using polymake, by the way. I haven't tried the version from hpc.sourceforge.net because if it breaks things, I don't know how hard it will be to uninstall.

[fbissey]

Replying to jhpalmieri:

I built and installed gcc 5.4.0 and got errors building libfplll and givaro:

ld: symbol(s) not found for architecture x86_64

Maybe I'm building gcc wrong?

That's a classic message. Usually something slightly wrong in the setup. A bigger chunk of the failure may jog my memory.

I'm not using polymake, by the way. I haven't tried the version from hpc.sourceforge.net because if it breaks things, I don't know how hard it will be to uninstall.

It will be a pain to uninstall. Well not too bad so long as you keep the original tarball to have the list of files installed. It also helps that I don't have much in /usr/local in the first place. But I hate things like that, that insists they need to be installed in /usr/local. Whenever I try homebrew, I specifically do not follow their recommendation about linking in /usr/local - an invitation to disaster if you have more than one prefixed environment around that you want to be able to switch to and from.

[jhpalmieri]

Here's more of the error message from givaro:

Undefined symbols for architecture x86_64:
  "std::ctype<char>::_M_widen_init() const", referenced from:
      Givaro::Rationel::Rationel(float, Givaro::Rationel::reduceFlag) in libgmppp.a(gmp++_rat_cstor.o)
      Givaro::Rationel::Rationel(double, Givaro::Rationel::reduceFlag) in libgmppp.a(gmp++_rat_cstor.o)
      Givaro::GivMMInfo::print(std::basic_ostream<char, std::char_traits<char> >&) const in libgivmemory.a(givaromm.o)
      Givaro::Rational::ratrecon(Givaro::Integer const&, Givaro::Integer const&, Givaro::Integer const&, bool) in libgivrational.a(givratreconstruct.o)
      Givaro::GivaroMain::DisplayVersion(std::basic_ostream<char, std::char_traits<char> >&) in libgivsystem.a(givinit.o)
      Givaro::Timer::print(std::basic_ostream<char, std::char_traits<char> >&) const in libgivsystem.a(givtimer.o)
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::swap(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
      Givaro::Indeter::operator=(Givaro::Indeter const&) in libgivpoly1.a(givindeter.o)
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)", referenced from:
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.21] in libgmppp.a(gmp++_rat_io.o)
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.51] in libgmppp.a(gmp++_int_misc.o)
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) [clone .isra.19] in libgivrational.a(givratcstor.o)
  "std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  "std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_istringstream()", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  "std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  "std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)", referenced from:
      Givaro::logp(Givaro::Integer const&, Givaro::Integer const&) in libgmppp.a(gmp++_int_misc.o)
  "std::__detail::_List_node_base::_M_unhook()", referenced from:
      Givaro::logp(Givaro::Integer const&, Givaro::Integer const&) in libgmppp.a(gmp++_int_misc.o)
  "std::basic_istream<char, std::char_traits<char> >& std::operator>><char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)", referenced from:
      Givaro::operator>>(std::basic_istream<char, std::char_traits<char> >&, Givaro::Indeter&) in libgivpoly1.a(givindeter.o)
  "VTT for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  "VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  "vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status

[jhpalmieri]

And from libpflll:

Undefined symbols for architecture x86_64:
  "std::ctype<char>::_M_widen_init() const", referenced from:
      fplll::shortest_vector_ex(fplll::ZZ_mat<__mpz_struct [1]>&, std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >&, fplll::SVPMethod, std::vector<double, std::allocator<double> > const&, int, fplll::EvaluatorMode, long long&, std::vector<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >, std::allocator<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > > > >*, std::vector<double, std::allocator<double> >*, std::vector<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >, std::allocator<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > > > >*, std::vector<double, std::allocator<double> >*, int) [clone .constprop.282] in fplll.o
      fplll::closest_vector(fplll::ZZ_mat<__mpz_struct [1]>&, std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > > const&, std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >&, int, int) in fplll.o
      fplll::EnumerationDyn<fplll::FP_NR<dpe_struct [1]> >::process_solution(double) in enumerate.o
      fplll::EnumerationDyn<fplll::FP_NR<long double> >::process_solution(double) in enumerate.o
      fplll::EnumerationDyn<fplll::FP_NR<double> >::process_solution(double) in enumerate.o
      fplll::FastEvaluator<fplll::FP_NR<__mpfr_struct [1]> >::eval_sol(std::vector<fplll::FP_NR<__mpfr_struct [1]>, std::allocator<fplll::FP_NR<__mpfr_struct [1]> > > const&, double const&, double&) in enumerate.o
      fplll::FastEvaluator<fplll::FP_NR<dpe_struct [1]> >::eval_sol(std::vector<fplll::FP_NR<dpe_struct [1]>, std::allocator<fplll::FP_NR<dpe_struct [1]> > > const&, double const&, double&) in enumerate.o
      ...
  "std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      ...
  "std::domain_error::domain_error(char const*)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::const_iterator::operator==(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::const_iterator const&) const in bkz_param.o
  "std::domain_error::domain_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.285] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.286] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.287] in bkz_param.o
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&>(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&) in bkz_param.o
      unsigned long nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::get_impl<unsigned long, 0>(unsigned long*) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
      ...
  "std::out_of_range::out_of_range(char const*)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::const_iterator::operator*() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
  "std::runtime_error::runtime_error(char const*)", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) [clone .constprop.727] in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_preprocessing(int, int, fplll::BKZParam const&) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) [clone .constprop.728] in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) [clone .constprop.731] in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::svp_preprocessing(int, int, fplll::BKZParam const&) in bkz.o
      ...
  "std::basic_ifstream<char, std::char_traits<char> >::basic_ifstream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
  "std::invalid_argument::invalid_argument(char const*)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      fplll::Pruner<fplll::FP_NR<double> >::measure_metric(std::vector<double, std::allocator<double> > const&) in pruner.o
      fplll::Pruner<fplll::FP_NR<double> >::measure_metric(std::array<fplll::FP_NR<double>, 1023ul> const&) in pruner.o
      fplll::Pruner<fplll::FP_NR<double> >::repeated_enum_cost(std::array<fplll::FP_NR<double>, 1023ul> const&) in pruner.o
      fplll::Pruner<fplll::FP_NR<double> >::greedy(std::array<fplll::FP_NR<double>, 1023ul>&) in pruner.o
      fplll::FP_NR<double> fplll::svp_probability<fplll::FP_NR<double> >(fplll::Pruning const&) in pruner.o
      fplll::FP_NR<double> fplll::svp_probability<fplll::FP_NR<double> >(std::vector<double, std::allocator<double> > const&) in pruner.o
      ...
  "std::invalid_argument::invalid_argument(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::expect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) in bkz_param.o
      void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&>(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&) in bkz_param.o
      unsigned long nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::get_impl<unsigned long, 0>(unsigned long*) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::expect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
      ...
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_erase(unsigned long, unsigned long)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long)", referenced from:
      fplll::strategy_full_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::expect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
      ...
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)", referenced from:
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.189] in bkz_param.o
      fplll::default_strategy_path[abi:cxx11]()  in bkz_param.o
      fplll::strategy_full_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      fplll::default_strategy[abi:cxx11]()  in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.285] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.287] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      ...
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()", referenced from:
      fplll::default_strategy_path[abi:cxx11]()  in bkz_param.o
      fplll::strategy_full_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      fplll::default_strategy[abi:cxx11]()  in bkz_param.o
  "std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::_Ios_Openmode)", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      ...
  "std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      ...
  "std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)", referenced from:
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_2reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_2reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_3reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_3reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_4reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_4reduce(ListPoint<long>*) in sieve_gauss.o
  "std::__detail::_List_node_base::_M_unhook()", referenced from:
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_2reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_3reduce_2reduce(ListPoint<__mpz_struct [1]>*, std::_List_iterator<ListPoint<__mpz_struct [1]>*>&) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_3reduce_2reduce(ListPoint<long>*, std::_List_iterator<ListPoint<long>*>&) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_2reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_4reduce_3reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_3reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_3reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      ...
  "std::__throw_bad_function_call()", referenced from:
      fplll::ExternalEnumeration<fplll::FP_NR<__mpfr_struct [1]> >::enumerate(int, int, fplll::FP_NR<__mpfr_struct [1]>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      fplll::ExternalEnumeration<fplll::FP_NR<double> >::enumerate(int, int, fplll::FP_NR<double>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      fplll::ExternalEnumeration<fplll::FP_NR<long double> >::enumerate(int, int, fplll::FP_NR<long double>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      fplll::ExternalEnumeration<fplll::FP_NR<dpe_struct [1]> >::enumerate(int, int, fplll::FP_NR<dpe_struct [1]>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
  "std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status

[fbissey]

Which version of OS X are you running?

[fbissey]

Successfully built with gcc-7.1 from hpc.sourceforge.net.

[jhpalmieri]

Replying to fbissey:

Which version of OS X are you running?

Sierra (10.12.5).

Successfully built with gcc-7.1 from hpc.sourceforge.net.

Me too.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d76da7b	Merge branch 'develop' into compiler
​a50d44f	bump config tarball
​31ccf12	Merge branch 'compiler' into clang-progress
​fb86df8	update configure tarball

[fbissey]

@jhpalmieri your problem with 5.4.0 from homebrew is very strange. I don't any of the tricks in gcc's spkg-install deal with something like that. It is like it is trying to use the libstdc++ from the wrong arch to link. That's utterly bizarre.

[jhpalmieri]

It's not homebrew, I just got a gcc tarball and essentially did ./configure; make; make install. Maybe some configure flags should be added.

[fbissey]

Replying to jhpalmieri:

It's not homebrew, I just got a gcc tarball and essentially did ./configure; make; make install. Maybe some configure flags should be added.

Possibly. That's one of these things that are hard to assess without being over your shoulder.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dfebfaf

simplify spkg-install for gfortran a bit

[kcrisman]

Testing with a from-scratch git clone on OS X 10.11.6, Xcode 7.3.1. So far, so good ...

checking for C compiler vendor... clang
checking for clang... clang

and I'm not seeing the usual message about building a small mpir to build gcc.

[kcrisman]

BUT

Found local metadata for gfortran-5.4.0
Attempting to download package gcc-5.4.0.tar.bz2 from mirrors
http://files.sagemath.org/spkg/upstream/gfortran/gcc-5.4.0.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//files.sagemath.org/spkg/ups$
http://mirrors-usa.go-parts.com/sage/sagemath/spkg/upstream/gfortran/gcc-5.4.0.$
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//mirrors-usa.go-parts.com/sa$
http://mirror.yandex.ru/mirrors/sage.math.washington.edu/spkg/upstream/gfortran$
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]

Etc.

And this makes sense, because http://files.sagemath.org/spkg/upstream/ doesn't have a gfortran directory. So I had to download it "by hand" from files.sagemath.org. Something along these lines would have to be fixed eventually. But for testing purposes at the least you should point this out in your instructions.

For now, that seemed to do it. Though

[gfortran-5.4.0] The following languages will be built: c,fortran

Does it need to build c?

[dimpase]

Replying to kcrisman:

For now, that seemed to do it. Though

[gfortran-5.4.0] The following languages will be built: c,fortran

Does it need to build c?

IMHO it cannot at the moment build Fortran only, it has to be C as well.

[jhpalmieri]

By the way, I reinstalled my own gcc-5.4.0, and now Sage builds and all tests pass. So it looks like the problems were on my end.

[kcrisman]

R was apparently successfully built in my case, so that is a very good sign.

[kcrisman]

Testing with a from-scratch git clone on OS X 10.11.6, Xcode 7.3.1. So far, so good ...

Sage successfully built, modulo needing to download those two files. Currently building doc but that should be unrelated. Shall I run doctests?

[fbissey]

Replying to kcrisman:

Testing with a from-scratch git clone on OS X 10.11.6, Xcode 7.3.1. So far, so good ...

Sage successfully built, modulo needing to download those two files. Currently building doc but that should be unrelated. Shall I run doctests?

yes please! Thanks for the feedback.

I had always assumed people would start from a tarball so yes I will add instructions shortly. If we go that route the gfortran folder in the upstream repo should link to the gcc one.

@ jhpalmieri That an external gcc-5.4.0 works is great news.

Considering some feedback on #22646 I may have to introduce some versioning check. xcode 7 apparently works, I don't know if someone can try xcode 6. But xcode 7 is probably a good lower bound on OS X. Now apparently clang 3.7 on fedora 23 is quite broken so on linux I am thinking of putting the lower bound at 3.8. But install details may vary from distro to distro - incredibly I think clang is too modular for its own good.

[dimpase]

Could we ​enable freebsd on this branch?

(this link is to the corresponding trivial commit from #22679)

It looks as if it's the only thing needed on top of the branch here to build on freebsd (well, still with a bit of manual intervention).

[kcrisman]

No errors but an awful lot of timeouts, though that said I haven't run doctests for ages on this computer. I'll try again with SAGE_TIMEOUT=1000 but I don't expect much.

[fbissey]

Replying to kcrisman:

No errors but an awful lot of timeouts, though that said I haven't run doctests for ages on this computer. I'll try again with SAGE_TIMEOUT=1000 but I don't expect much.

That's unusual. Is the machine overloaded? Of course that could be down to one single critical element with a fault causing the timeouts.

[kcrisman]

No errors but an awful lot of timeouts, though that said I haven't run doctests for ages on this computer. I'll try again with SAGE_TIMEOUT=1000 but I don't expect much.

That's unusual. Is the machine overloaded? Of course that could be down to one single critical element with a fault causing the timeouts.

I think it just couldn't handle parallel testing of some files - everything that didn't pass was over 100s when done individual and was fine. The only thing that didn't actually pass at some point was

sage -t src/sage/repl/configuration.py
**********************************************************************
File "src/sage/repl/configuration.py", line 14, in sage.repl.configuration
Failed example:
    'In [1]: [False, True]' in output
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of   5 in sage.repl.configuration
    [18 tests, 1 failure, 31.96 s]

but this passed three times in a row when done individually so perhaps some obscure race condition. I say that this means we can use clang on my box, anyway.

[dimpase]

this works nicely on FreeBSD, with a small patch on top of this branch, see #22679. (few failing doctests, a bit more than on OSX).

[dimpase]

On Gentoo Linux with clang (4.0.0), there are C++ problems, no matter what -stdlib= is.

• -stdlib=libstdc++: in giac there are problems with hash_map and unordered_map - none work. Not sure it's gentoo or giac bug, or both... While the rest of the packages appear to build, this one does not look easy to tackle without the author. Another reason to stick to libc++, for the time being at least.

• -stdlib=libc++: some C++ problems (do not look to hard to fix) in eclib. More to come (still compiling more packages, to see what else is iffy)

[fbissey]

Replying to dimpase:

On Gentoo Linux with clang (4.0.0), there are C++ problems, no matter what -stdlib= is.

• -stdlib=libstdc++: in giac there are problems with hash_map and unordered_map - none work. Not sure it's gentoo or giac bug, or both... While the rest of the packages appear to build, this one does not look easy to tackle without the author. Another reason to stick to libc++, for the time being at least.

• -stdlib=libc++: some C++ problems (do not look to hard to fix) in eclib. More to come (still compiling more packages, to see what else is iffy)

Gentoo but I am still on clang-3.9.1.

There are indeed some issues with eclib. Apart from the tolerance issue one the test executable segfault for an unclear reason in a C++ header from clang's libstdc++. The issue doesn't appear to exist on OS X - where xcode is based on clang 3.8. eclib could gain to completely switch to c++11. It wouldn't fix all the issues but it would come out cleaner. The issues in eclib don't prevent sage test suite to fully pass.

giac, yes lots of warnings but apart from the funny test where the order of the solution flip between gcc and clang everything passes. I must say that I already have a bad taste in my mouth from hash_map/unordered_map in brial. I think hash_map should get the cold shoulder.

Now for something a little bit different. The branch in its current form will let you try to compile sage with the intel compiler (C/C++ not fortran). So far I only needed two fixes aside from the one already needed for clang. sqlite (because icc fakes gcc too much and sqlite didn't think of it) and openblas (known issue fixed in master but that missed the 0.2.19 release).

[dimpase]

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors. How does one submit patches for giac?

[fbissey]

Replying to dimpase:

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors.

OK, makes sense. We probably want to switch most of these to std=c++11 anyway. I am writing a pull request for eclib to change its default and drop the stuff that c++11 doesn't like (keyword "register").

How does one submit patches for giac?

You have to raise an issue on their forum at ​http://xcas.e.ujf-grenoble.fr/XCAS , note that me and debian are already queuing for some other changes related to gui and the use of X libraries.

[dimpase]

Replying to fbissey:

Replying to dimpase:

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors.

Unintentionally, I built with -stdlib=libstdc++; apparently in my configuration I need an explicit -stdlib=libc++ in CXXFLAGS.

Anyhow, ptestlong fails just one doctest, due to numerical noise:

File "src/sage/libs/lcalc/lcalc_Lfunction.pyx", line 188, in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
Failed example:
    L.hardy_z_function(0)
Expected:
    0.7939675904771...
Got:
    0.793967590477206 - 4.03312243040125e-17*I

I'd say this configuration is good to go. I'll test with -stdlib=libc++, too. I guess some GNUisms might pop up...

OK, makes sense. We probably want to switch most of these to std=c++11 anyway. I am writing a pull request for eclib to change its default and drop the stuff that c++11 doesn't like (keyword "register").

Yes, this is a good idea.

How does one submit patches for giac?

You have to raise an issue on their forum at ​http://xcas.e.ujf-grenoble.fr/XCAS , note that me and debian are already queuing for some other changes related to gui and the use of X libraries.

It seems I need to jump through a French-only hoop to subscribe to that forum :-)

[fbissey]

Replying to dimpase:

Replying to fbissey:

Replying to dimpase:

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors.

Unintentionally, I built with -stdlib=libstdc++; apparently in my configuration I need an explicit -stdlib=libc++ in CXXFLAGS.

Anyhow, ptestlong fails just one doctest, due to numerical noise:

File "src/sage/libs/lcalc/lcalc_Lfunction.pyx", line 188, in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
Failed example:
    L.hardy_z_function(0)
Expected:
    0.7939675904771...
Got:
    0.793967590477206 - 4.03312243040125e-17*I

That's funny.

fbissey@moonloop ~/sandbox/git-fork/sage-icc $ ./sage -t --long src/sage/libs/lcalc/lcalc_Lfunction.pyx 
too many failed tests, not using stored timings
Running doctests with ID 2017-06-29-20-00-14-17cf5c10.
Git branch: clang-progress
Using --optional=mpir,python2,sage
Doctesting 1 file.
sage -t --long src/sage/libs/lcalc/lcalc_Lfunction.pyx
**********************************************************************
File "src/sage/libs/lcalc/lcalc_Lfunction.pyx", line 188, in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
Failed example:
    L.hardy_z_function(0)
Expected:
    0.7939675904771...
Got:
    0.793967590477208 - 4.03312243040124e-17*I
**********************************************************************
1 item had failures:
   1 of  15 in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
    [103 tests, 1 failure, 1.10 s]

Yes, this is my icc build. I think it deserves a separate numerical noise ticket.

I'd say this configuration is good to go. I'll test with -stdlib=libc++, too. I guess some GNUisms might pop up...

OK, makes sense. We probably want to switch most of these to std=c++11 anyway. I am writing a pull request for eclib to change its default and drop the stuff that c++11 doesn't like (keyword "register").

Yes, this is a good idea.

I should surprise John with that PR soon.

How does one submit patches for giac?

You have to raise an issue on their forum at ​http://xcas.e.ujf-grenoble.fr/XCAS , note that me and debian are already queuing for some other changes related to gui and the use of X libraries.

It seems I need to jump through a French-only hoop to subscribe to that forum :-)

Was it all French? I can't remember :)

[dimpase]

with clang 4.0.1 and -stdlib=libc++ -std=c++11 for eclib to build I need

diff --git a/libsrc/eclib/interface.h b/libsrc/eclib/interface.h
index b8eff26..20c9c98 100644
--- a/libsrc/eclib/interface.h
+++ b/libsrc/eclib/interface.h
@@ -159,10 +159,7 @@ inline int is_approx_zero(const RR& x)
   // cout<<"is_approx_zero() returns "<<(x.mantissa()<power2_ZZ(-n))<<endl;
   return abs(x.mantissa())<power2_ZZ(-n);
 }
-} // namespace NTL
 #ifdef _LIBCPP_VERSION
-namespace std {
-inline namespace __1 {
 inline bool isinf(const RR& z) {
   return false;
 }
@@ -182,9 +179,8 @@ inline RR fmax(const RR& x, const RR& y)
 {
   return x < y ? y : x;
 }
-}
-}
 #endif
+}
 
 #include <complex>
 typedef complex<RR> CC;

which basically moves isinf, insnan, etc to the namespace NTL. Does this break clang on OSX?

[fbissey]

Replying to dimpase:

which basically moves isinf, insnan, etc to the namespace NTL. Does this break clang on OSX?

I will try and see what it does here.

[fbissey]

Makes no difference whatsoever on OS X. But it reminded me that I had the same weird segfault on OS X as I have on linux with clang 3.9.1 (in the testsuite).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5434c70	Merge branch 'develop' into compiler
​f565739	bump config tarball
​53f9c4d	Merge branch 'compiler' into clang-progress
​7a9ab6b	checksum for this branch's configure tarball

[fbissey]

Tried clang-4.0.1 on linux still the same stuff happening during the testsuite. Will see what happens on OS X.

The patch for eclib will need its own ticket, with a matching PR upstream and needs testing with gcc.

The lcalc doctest should also get its own ticket.

[dimpase]

Replying to fbissey:

Tried clang-4.0.1 on linux still the same stuff happening during the testsuite. Will see what happens on OS X.

The patch for eclib will need its own ticket, with a matching PR upstream and needs testing with gcc.

The lcalc doctest should also get its own ticket.

On Linux clang 4.0.1 with libc++ gives a weird linking error on gfan (another old C++ worry).

It took me a while to notice that CXXFLAGS is not used during compilation of C++. (and in my setup -stdlib=libc++ is not the default, the default is libstdc++, so CXXFLAGS are really needed!) It's an error in gfan's makefile that somehow went unnoticed. The following (a patch to Sage's patch :-)) fixes it:

diff --git a/build/pkgs/gfan/patches/Makefile.patch b/build/pkgs/gfan/patches/Makefile.patch
index 648fa56902..2c4e1d2d3f 100644
--- a/build/pkgs/gfan/patches/Makefile.patch
+++ b/build/pkgs/gfan/patches/Makefile.patch
@@ -60,7 +60,7 @@ index 0892969..c626eb9 100644
        $(CXX) -c $<
  .cpp.o:
 -      $(CXX) $(CFLAGS) -c $<
-+      $(CXX) $(CFLAGS_LOCAL) $(CPPFLAGS) -c $<
++      $(CXX) $(CXXFLAGS_LOCAL) $(CPPFLAGS) -c $<
  .C.o:
        $(CXX) -c $<
  # wget http://ftp.sunet.se/pub/gnu/gmp/gmp-4.2.2.tar.gz

Should we have a separate ticket for this?

[fbissey]

Replying to dimpase:

Should we have a separate ticket for this?

Yes.

[jdemeyer]

Bikeshed: I prefer configure messages to be short. 3 lines just to say that gfortran needs to be installed is too much:

AC_MSG_NOTICE([GFORTRAN will be built.])
AC_MSG_NOTICE([Your Fortran compiler isn't GCC (GNU Fortran) or])
AC_MSG_NOTICE([you don't have one.])

[jdemeyer]

Would it be possible to treat gfortran as a completely normal package? That means, do not build it during all-toolchain, but just add it explicitly as dependency to the packages that need it. That would speed up compilation since gfortran could be built in parallel with other packages (the toolchain is built serially, so it should contain as few packages as possible).

[jdemeyer]

Please do create new tickets for further issues regarding specific packages. This ticket is nicely self-contained, dealing only with the basic Sage configure/build system.

[dimpase]

Replying to jdemeyer:

Would it be possible to treat gfortran as a completely normal package? That means, do not build it during all-toolchain, but just add it explicitly as dependency to the packages that need it. That would speed up compilation since gfortran could be built in parallel with other packages (the toolchain is built serially, so it should contain as few packages as possible).

Sure, why not - except that then all the packages that need it must get the corresponding dependency, on the package gfortran. (it's relatively few: numpy, scipy, R, anything esle?)

[fbissey]

Replying to jdemeyer:

Bikeshed: I prefer configure messages to be short. 3 lines just to say that gfortran needs to be installed is too much:

AC_MSG_NOTICE([GFORTRAN will be built.])
AC_MSG_NOTICE([Your Fortran compiler isn't GCC (GNU Fortran) or])
AC_MSG_NOTICE([you don't have one.])

Fair enough. I'll try to be more terse. The previous message was annoying me as potentially misleading: "Your fortran compiler is not GCC..." it creates the perception you have a fortran compiler even when you have none.

Dima has a fair point about dependency on gfortran. But the only dependency needed is for blas since all the package needing fortran need it for lapack.

[kcrisman]

Dima has a fair point about dependency on gfortran. But the only dependency needed is for blas since all the package needing fortran need it for lapack.

Oh, R doesn't need it directly? Certainly some R packages do need fortran directly, though of course supporting every R package in existence might be tricky.

[jdemeyer]

To be safe, we could (and should) add gfortran explicitly as a dependency of R. I guess the main point of François was that it's not a problem if we forget one package which needs Fortran because of BLAS.

[fbissey]

R depends on lapack, although you are right some packages depends on fortran directly. The same is true for scipy, it depends on lapack and uses some fortran code directly. I am fairly sure numpy only depends on lapack/blas. Some optional packages would need to be checked (coinor uses lapack).

If we include the external dependencies required by cvxopt 1.1.9 some of them may depend directly on fortran.

[fbissey]

A practical question about putting gfortran in dependencies. If you use a system gfortran, a logs/pkg/gfortran-x.log is not produced. Same as with gcc. How does the dependency system knows that gfortran has been installed? I am guessing I should look at yasm for an example.

[fbissey]

Somehow it must work packages that need to depend on (g)fortran - add any packages that missing to the list:

• openblas
• atlas
• R
• scipy
• numpy (linking with lapack means that numpy needs to at least know about your fortran compiler apparently)

On the top of my head none of the other packages in standard, optional or experimental needs fortran. But I am unfamiliar with some of those packages in optional and experimental.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5326571

make gfortran a regular dependency. Simplify message in configure.

[jdemeyer]

See 325

[jdemeyer]

Replying to fbissey:

A practical question about putting gfortran in dependencies. If you use a system gfortran, a logs/pkg/gfortran-x.log is not produced. Same as with gcc. How does the dependency system knows that gfortran has been installed? I am guessing I should look at yasm for an example.

The configure variable need_to_install_gfortran adjusts the generated build/make/Makefile. And indeed yasm is a good example.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f4d6748	Change the setting of compilers in the order: environment, sage installed compilers, configured compilers.
​eb6f027	fix objective compiler detection. Don't try to negate using the macro. Bad syntax that no one seem to have noticed.
​4c41390	Regenerate configure tarball
​38273c6	Remove bits specific to outdated xcode toolchain
​ab61da6	update configure tarball
​5820812	Merge branch 'develop' into compiler and bump configure tarball
​e89c07c	Merge branch 'compiler' into clang-progress and update configure tarball

[dimpase]

I noticed that the optional package boost does not build ob FreeBSD, and gets built with gcc on Linux/clang/libc++ even if CC and CXX are specified.

It seems we need to fix spkg-install there to use --toolset $CC option. As well, passing -std=... has to be done explicitly, it seems; does this mean that the toplevel Sage configure also needs something like CXXLIB= option.

[fbissey]

Good catch, I'll have a look later today. I don't like the idea of adding CXXLIB that's very specific. But we can sort out some details later. boost has a, shall we say, very peculiar build system, only used by boost. The boost dev would you to use it for your project involving boost but it doesn't seem to get much traction (thanks goodness).

[fbissey]

Hum... that will be interesting, it will need its own ticket of course. From bootstrap.sh

# Determine the toolset, if not already decided
if test "x$TOOLSET" = x; then
  guessed_toolset=`$my_dir/tools/build/src/engine/build.sh --guess-toolset`
  case $guessed_toolset in
    acc | darwin | gcc | como | mipspro | pathscale | pgi | qcc | vacpp )
    TOOLSET=$guessed_toolset
    ;;
    
    intel-* )
    TOOLSET=intel
    ;;
    
    mingw )
    TOOLSET=gcc
    ;;
    
    sun* )
    TOOLSET=sun
    ;;
    
    * )
    # Not supported by Boost.Build
    ;;
  esac
fi

clang is not a defined toolset, may be we'd end up with darwin (no), it should probably piggy back on gcc. May be we need an update to boost. We could possibly leave the toolset as gcc for now and use a user-config.jam file with our settings.

It must already do funny things on OS X since in the absence of toolset, darwin is used and probably default to clang (unless they are counting on gcc to be clang, it is not impossible considering the rest of the mess).

The build system is complex and dealing with it in a systematic way for other vendor (say intel) is difficult.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​79d0963	Remove compiler selecting logic from sage-env-config and put it back in sage-env.
​d20a8e7	Merge branch 'compiler' into clang-progress

[fbissey]

OK I am wrong for clang support. It is only guessed for freebsd - and then according to what I quoted it falls in the unsupported category, but that doesn't stop anything. And provided I setup things right (which I hadn't before) it builds on linux with toolset=clang.

So on looking closer at the current situation (i.e. without this ticket), on OS X we probably currently build with clang (cc seem to be used) so it is probably on shaky ground with the rest of sage right now (c++ runtime incompatibilities).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1015a1f	Merge branch 'master' into compiler and configure tarball bump
​40bd88a	Merge branch 'compiler' into clang-progress

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c4fce28	fix missing "$" in sage-env
​bbee7b8	Merge branch 'develop' into compiler and bump configure tarball
​eaf6473	Merge branch 'compiler' into clang-progress and regenerate configure tarball

[fbissey]

Note that as of sage 8.1.beta0 this ticket can be tested out of the box on linux.

[dimpase]

there is a mismatch with the value in checksums.ini

$ md5sum configure-232.tar.gz 
4aa680b6dfc4c372872a1a5ee4ec781b  configure-232.tar.gz

vs in checksums.ini

md5=0ab186b7307444b0f2e7f47413eb5ba2

[fbissey]

Strange I have the right checksum. I replaced the link in the description as if you cut and paste the address that was there you'll download a web page instead of the tarball. I am wondering if that's you problem. It should be a direct link now.

[dimpase]

Replying to fbissey:

Strange I have the right checksum. I replaced the link in the description as if you cut and paste the address that was there you'll download a web page instead of the tarball. I am wondering if that's you problem.

Oh, indeed, it's some http weirdness that creates a file named configure-232.tar.gz but it is actually not a gzipped tarfile...

Trying the branch on Linux/clang now.

[dimpase]

OK, this builds and seems to work on Gentoo/clang/libc++, if I add CXXFLAGS="-stdlib=libc++. Without the latter (i.e. with the default libstdc++ instead) it does not work with giac and probably more.

[dimpase]

One has however fix the problem that libgiac pulls in libstdc++.so as well as libc++.so, and this leads to giac basically being non-functional. But this is for another ticket.

[jhpalmieri]

By the way, I've been trying this on OS X with each new Sage release, and it's worked well each time. (This is with my own gfortran installed, by the way.)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e485a75	Merge branch 'develop' into compiler and bump configure tarball
​6f17c02	Merge branch 'compiler' into clang-progress

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ceb3bf5

conform to new packaging in place since 8.1.beta1

[dimpase]

Could we have a rebase here?

[fbissey]

How urgent is it? The main reason for the need to rebase is that I keep providing a configure tarball which is updated every beta (I think). Otherwise the branch wouldn't need rebasing so often.

[dimpase]

Perhaps it makes sense to require autoconf instead, for the purposes of this ticket going forward at least. We can put configure tarball back once it's ready to go.

[fbissey]

OK, it will have to be the same with #22646 on which it depends. I have stuff to put in #22646, I will proceed with the autoconf requirement from there. But I am afraid it will have to wait until I go back to work tomorrow morning in my time zone.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3772ae5	Merge branch 'develop' into compiler
​a8c8904	Add some documentation and update configure tarball
​07bfc5a	Merge branch 'compiler' into clang-progress

[fbissey]

OK it will still need rebasing until #22646 is merged because I still the tarball there for now. We are closing in on the review for that one so it may be necessary. But this ticket doesn't provide a tarball now.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​99eb31f	Minor edits from John Palmieri's feedback
​d8cb828	Merge branch 'compiler' into clang-progress

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0ec453b

Merge branch 'develop' into clang-progress

[dimpase]

Where is the gfortran tarball?

[fbissey]

Replying to dimpase:

Where is the gfortran tarball?

No gfortran tarball, gcc's tarball is used.

[dimpase]

Replying to fbissey:

Replying to dimpase:

Where is the gfortran tarball?

No gfortran tarball, gcc's tarball is used.

the spkg-install of gfortran package is too dumb to know this, and without gfortran or gcc, and CC set to clang this ends up in an error downloading a non-existent tarball...

[fbissey]

This shouldn't happen, spkg-install doesn't know the name of the tarball, package-version.txt does. Can you show me the log?

[dimpase]

it was already mentioned here; namely, there is no gfortran/ directory on the servers:

Found local metadata for gfortran-5.4.0
Attempting to download package gcc-5.4.0.tar.bz2 from mirrors
Downloading the Sage mirror list
Searching fastest mirror
  149ms: http://files.sagemath.org/
...
    6ms: http://www.mirrorservice.org/sites/www.sagemath.org/
Fastest mirror: http://www.mirrorservice.org/sites/www.sagemath.org/
http://www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/gfortran/gcc-5.4.0.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/gfortran/gcc-5.4.0.tar.bz2'
...

[fbissey]

Yes it says in the description that you have to make sure the gcc tarball is already downloaded. The release manager needs to create a link gfortran -> gcc when merging the ticket. This is also in the description.

[dimpase]

Oops, sorry, I was too quick regarding the tarball. Anyway, in this configuration on FreeBSD one is hit by the usual platform specific complex numbers support SNAFU.

[gfortran-5.4.0] libtool: compile:  /usr/home/dima/Sage/porting/sage/local/var/tmp/sage/build/gfortran-5.4.0/gcc-build/./gcc/xgcc -B/
usr/home/dima/Sage/porting/sage/local/var/tmp/sage/build/gfortran-5.4.0/gcc-build/./gcc/ -B/usr/home/dima/Sage/porting/sage/local/x86
o -c ../../../src/libgfortran/runtime/memory.c  -fPIC -DPIC -o .libs/memory.o
[gfortran-5.4.0] In file included from ../../../src/libgfortran/libgfortran.h:50:0,
[gfortran-5.4.0]                  from ../../../src/libgfortran/fmain.c:4:
[gfortran-5.4.0] ../../../src/libgfortran/c99_protos.h:311:14: error: expected declaration specifiers or '...' before '(' token
[gfortran-5.4.0]  extern float cargf (float complex);
[gfortran-5.4.0]               ^
[gfortran-5.4.0] ../../../src/libgfortran/c99_protos.h:311:14: error: expected declaration specifiers or '...' before '(' token
[gfortran-5.4.0]  extern float cargf (float complex);
[gfortran-5.4.0]               ^
[gfortran-5.4.0] ../../../src/libgfortran/c99_protos.h:316:15: error: expected declaration specifiers or '...' before '(' token
[gfortran-5.4.0]  extern double carg (double complex);
[gfortran-5.4.0]                ^

IMHO we should abandon this particular FreeBSD configuration, it's too much trouble, as gfortran is readily available on the system.

[jhpalmieri]

With the latest release of Xcode (version 9.0) on OS X, eclib now fails to build for me (using Sage 8.1.beta5 plus this branch). ​See the log file.

[fbissey]

You are ahead of me. I am surprised by the problem in eclib, it looks like the new apple clang is based on 3.9 which I have been initially working with on linux. And everything is fine with 4.0 both on linux and OS X. I am not sure if the problem is really in eclib or ntl.

[jhpalmieri]

Can you reproduce the problem? (You may not want to install Xcode 9.0, of course.)

[fbissey]

Replying to jhpalmieri:

(You may not want to install Xcode 9.0, of course.)

Too late by a couple of hours ^_^. I am a bit busy right now (and it is lunch time too) so it will have to wait.

[fbissey]

Ticket upstream at ​https://github.com/JohnCremona/eclib/issues/28

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​83815b2	Merge branch 'develop' into clang-progress
​40d3214	Merge branch 'develop' into clang-progress

[jdemeyer]

Shouldn't gfortran be an order-only dependency (after the | instead of before)?

[jdemeyer]

Various comments about gfortran/spkg-install:

1. Replace

   if [ -a "$SAGE_LOCAL"/bin/gcc ]; then
       echo >&2 "gcc appear to already be installed ... exiting"
       exit 1
   fi
   

by

if [ -x "$SAGE_LOCAL"/bin/gcc ]; then
    echo >&2 "Error: gcc is already installed"
    exit 1
fi

2. First doing cd src and then immediately cd .. is silly: drop that.

3. This isn't needed, the usual build dependency tracking will handle it:

   # Force re-configuration: the next time that "make" is run, we don't
   # want GFORTRAN to be built again, see Trac #19324
   touch "$SAGE_ROOT/configure"
   

[jdemeyer]

In configure.ac, I would remove AC_MSG_NOTICE([GFORTRAN wasn't found, so it will be built.]): there is no need to mention every package that will be built.

[fbissey]

Thanks for the comments, I haven't had good ones in some time :) About (3), I picked that bit from gcc's spkg-install - in a lot of ways gfortran is "gcc lite" - does that mean that this could be dropped from gcc's spkg-install (in a different ticket)?

[jdemeyer]

The Force re-configuration part is related to this block in GCC:

# Force re-installation of gmp, mpir, mpfr and mpc with the GCC we just built.
cd "$SAGE_SPKG_INST"
rm -f gmp-* mpir-* mpfr-* mpc-*

So it is needed for GCC but not for GFortran.

[jdemeyer]

Relevant ticket that I just created: #23919. It disables GNU extensions while compiling the Sage library. This exposes in particular the known issue with lcalc (#23341).

[jdemeyer]

Speaking of which, does this ticket depend on #23341? Does lcalc build with Clang currently?

[dimpase]

Replying to jdemeyer:

Speaking of which, does this ticket depend on #23341? Does lcalc build with Clang currently?

There are many clangs around. I think the one that does not work with lcalc is clang 4.x.y, for some ranges of x and y. clang 5.0.0 does compile the current clang, and it did work with clang 3.u.v for some u and v, as well.

And there is also an issue of the standard C++ library, libstdc++ (from GCC project) vs libc++ (from Clang project); they differ quite a bit. libstdc++ with clang seems to be a deadly mix...

Oh, and there also different runtime libs on Linux that may be used...

[fbissey]

I and a few other people have been compiling successfully sage, including lcalc, with xcode 8 on OS X, clang-4.0 (not xcode) on OS X, clang 3.9 and 4.0 on linux (Gentoo). xcode 9 and some other platforms and clang installs have problems with eclib (fixed in github master but no new release before October John said).

[jhpalmieri]

Regarding eclib, adding a file containing

libsrc/eclib/interface.h

@@ -161 +161 @@
 }
 } // namespace NTL
 #ifdef _LIBCPP_VERSION
-namespace std {
-inline namespace __1 {
+namespace NTL {
 inline bool isinf(const RR& z) {
   return false;
 }
@@ -183 +182 @@
   return x < y ? y : x;
 }
 }
-}
 #endif
 
 #include <complex>

to the patches directory allows it to build for me on OS X with Xcode 9.0.

[fbissey]

I guess we can spin a ticket with that one fix to eclib but just until we have the new release. The new release will properly default to c++11 and contain a number of minor fixes for clang. Ok some of the bugs revealed by eclib's testsuite (mostly fixed) are not so minor but they are not breaking sage's testsuite.

[jhpalmieri]

From my point of view, it depends on how long it will be before a new release of eclib. It would be nice to be able to build Sage in the meantime.

[fbissey]

The problem is only with some clang configurations, the most important one being xcode 9 for which we'll need a gcc bump as well. But yes let's put that in a new ticket until next month when we can have a release.

[fbissey]

Replying to jdemeyer:

Shouldn't gfortran be an order-only dependency (after the | instead of before)?

Tricky. Change in gcc version may or may not be accompanied by incompatible changes in fortran runtime. I should double-check on that since a change in libgfortran does mean that things need to be rebuilt. From ​https://gcc.gnu.org/wiki/GFortran/News it appears all the versions from 4.9.1 are fully ABI compatible (although .mod files have changed at gcc-5 but we don't have any of those I think) so far. It is not excluded that some incompatibility will be introduced in some future release.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6902166	Simplify gfortran spkg-install a bit. Thanks to Jeroen Demeyer for the feedback on this.
​afca2b6	no need to be so verbose about gfortran.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​235c09e	Including upstream PR 29 to compile with xcode 9 until we have a new release.
​48462ea	Merge branch 'eclib-xcode9' into clang-progress

[dimpase]

gfortran package does not respect SAGE_INSTALL_GCC=no, while it should? Or, perhaps, there should be SAGE_INSTALL_GFORTRAN variable overriding gfortran installation

(I'm trying to build with Intel's ifort compiler, and the 1st obstacle is the need to disable gfortran building...)

[fbissey]

Yes the current setup doesn't allow for alternative fortran compiler at all. gfortran or nothing. So it is not just a matter of not installing, and yes it would need its own variable.

What you really need to change is remove lines 477-479 of configure.ac. These lines are triggering gfortran build if GFC is not yes. GFC identifying gnu fortran. Look for GFC.

[jdemeyer]

Replying to fbissey:

Replying to jdemeyer:

Shouldn't gfortran be an order-only dependency (after the | instead of before)?

Tricky. Change in gcc version may or may not be accompanied by incompatible changes in fortran runtime.

I see. It is not just about the compiler, but also about the runtime library gfortran.so.

[fbissey]

Replying to jdemeyer:

I see. It is not just about the compiler, but also about the runtime library gfortran.so.

Exactly.

[dimpase]

Replying to fbissey:

Yes the current setup doesn't allow for alternative fortran compiler at all. gfortran or nothing. So it is not just a matter of not installing, and yes it would need its own variable.

What you really need to change is remove lines 477-479 of configure.ac. These lines are triggering gfortran build if GFC is not yes. GFC identifying gnu fortran. Look for GFC.

Thanks. I have opened #23926 to track the work on using other Fortran compilers.

[dimpase]

Does this need #23898 on OSX with Xcode 9? I gather yes, as otherwise gfortran cannot be built.

[jhpalmieri]

Replying to dimpase:

Does this need #23898 on OSX with Xcode 9? I gather yes, as otherwise gfortran cannot be built.

Yes, I guess it does need #23898, or at least the gfortran build fails without it. I haven't tried with it yet.

[jhpalmieri]

Now I've tried with and without #23898, and I can confirm that an upgraded GCC is necessary to build gfortran.

[dimpase]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​235c09e	Including upstream PR 29 to compile with xcode 9 until we have a new release.
​48462ea	Merge branch 'eclib-xcode9' into clang-progress

with progress made on #23922, the PR29 eclib patch must be removed.

[dimpase]

Can we decouple fortran issues from this ticket, have it done in this reduced form, and open a new one to get gfortran package in shape? (E.g. for systems that have fortran, e.g. FreeBSD, or perhaps OSX with gfortran from some non-Sage source, fortran is not relevant).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1c5c985

Merge branch 'develop' into clang-progress

[fbissey]

Replying to dimpase:

Can we decouple fortran issues from this ticket, have it done in this reduced form, and open a new one to get gfortran package in shape? (E.g. for systems that have fortran, e.g. FreeBSD, or perhaps OSX with gfortran from some non-Sage source, fortran is not relevant).

I am not sure I am very eager about this but this can definitely be done. If you think it can be merged quickly, I can separate the gfortran bits. I guess that will help with your "other fortran" project.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2f85d6c

Merge branch 'develop' into clang-progress

[fbissey]

Putting this back in needs review. It is a bit late for sage 8.1 I think, but if I think everything is in place for it to be usable and merged at the beginning of 8.2.

The biggest obstacles are the failing testsuite of eclib and giac. I should check the new giac-1.4.x series, may be the flipping behavior has been fixed - the patch for pari over 2.7 is not needed in that version (admittedly I didn't check compiling against pari 2.7, they may just have changed what they are compiling against internally and fixed accordingly).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1119be3

Remove eclib patch. It conflict with the new merged version of eclib

[dimpase]

IMHO as long as this is not yet used to build Sage for distributing, we should not worry about doctests failing due to small numerical noise or similar issues (speaking about eclib and giac, that is).

[dimpase]

Works for me, on FreeBSD and on OSX. I think it has been reviewed a lot already.

[jdemeyer]

Replying to dimpase:

we should not worry about doctests failing due to small numerical noise or similar issues

Of course we should. Doctests shouldn't fail.

In any case, I'd like to test this ticket more before merging it. So far, I have been mostly commenting on the diff without actually testing.

[kcrisman]

we should not worry about doctests failing due to small numerical noise or similar issues

Of course we should. Doctests shouldn't fail.

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind. This is the 416th comment so I think if it builds, it builds. Then open a new ticket for "passes tests".

[dimpase]

Replying to kcrisman:

we should not worry about doctests failing due to small numerical noise or similar issues

Of course we should. Doctests shouldn't fail.

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind. This is the 416th comment so I think if it builds, it builds. Then open a new ticket for "passes tests".

Yes, that's the right thing to do here, I think. We will open up a follow-up ticket once we have full OSX results (and perhaps Linux results?) for this one.

[jhpalmieri]

Just to clarify, the issue is not failing doctests (at least for me, make ptestlong passes all tests), but Sage packages failing their test suites.

[fbissey]

That's exactly where we are:

• sage doctests pass
• two standard packages (that would be checked by SAGE_CHECK=yes) are failing their testsuite
• optional packages test suite: unknown but for a few.

Of the two standard packages failing their testsuite:

• eclib: some results need careful checking, some may be fixed by having some tolerance in the tests.
• giac: it looks like the problem would be solved by going to 1.4.9-x - I tried yesterday.

[fbissey]

For the giac upgrade, #24174 is ready for review.

[jhpalmieri]

I tried the new giac package from #24174 plus the branch from this ticket, and it passes its test suite for me, too. (This is on two separate OS X machines, one with the current OS and Xcode, one with the previous version of each.)

[fbissey]

Only issue with giac may be the optional giacpy_sage, but I think it is already currently broken. Other than that and eclib it would be great if we could push this for 8.2.beta0.

[jhpalmieri]

The eclib (autocorrected to "eclair" which sounds good) issue still looks like numerical noise to me – ​here is my log file – but other people should check. In any case, I am happy with this. I will let Jeroen test more.

[fbissey]

Replying to jhpalmieri:

The eclib (autocorrected to "eclair" which sounds good) issue still looks like numerical noise to me – ​here is my log file – but other people should check. In any case, I am happy with this. I will let Jeroen test more.

That's because you stop after the first file to "error". If you add "-k" to the make command in spkg_check you'll find some other case that are not so obvious. Still the pure numerical noise could easily be fixed by some subtle C++ formatting and should be done. See ​https://github.com/JohnCremona/eclib/issues/19#issuecomment-315219838 for a more complete list of errors (the tmrank one has been fixed) and I am worried about the stuff generated by telog where some of the signs are flipped.

[jhpalmieri]

With the patch

build/pkgs/eclib/spkg-check

@@ -11 +11 @@
 
 echo
 echo "Now running eclib's test suite..."
-$MAKE check
+echo "Running $MAKE -k check"
+$MAKE -k check
 if [ $? -ne 0 ]; then
     echo >&2 "Error: eclib's test suite failed to pass."
     exit 1

I see numerical noise plus the following one. Is this the telog error you mentioned?

59,61c59,61
< Periods: [w_1,w_2] = [(-0.51317082433770156405898300169308452073251780098017e-48,-0.63277902985143330237168013012161275230207585428624),(2.0131613909534741204308876218764326959565176646346,-0.31638951492571665118584006506080637615103792714312)]
< tau       = (0.49999999999999999999999999999999999999999999999742,3.1814603455271474113187811974948516017119213527377) (abs(tau)=3.220510818202869536114214880382970192586945735383)
< w_R = (4.0263227819069482408617752437528653919130353292697,0)	w_IR = (2.0131613909534741204308876218764326959565176646351,0.31638951492571665118584006506080637615103792714312)
---
> Periods: [w_1,w_2] = [(0,0.6327790298514333023716801301216127523020758542864),(-2.0131613909534741204308876218764326959565176646351,-0.3163895149257166511858400650608063761510379271432)]
> tau       = (-0.5,3.1814603455271474113187811974948516017119213527374) (abs(tau)=3.2205108182028695361142148803829701925869457353831)
> w_R = (4.0263227819069482408617752437528653919130353292703,0)	w_IR = (2.0131613909534741204308876218764326959565176646351,0.3163895149257166511858400650608063761510379271432)

[fbissey]

Yes. Some of it is still numerical noise plus sign flip-flopping.

[jdemeyer]

Replying to kcrisman:

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind.

That was for supporting "new" systems for which there was previously no support. Here, we are talking about changing things for a system on which Sage already "passes tests". So going from "passes tests" to "builds" would be a regression.

[dimpase]

Replying to jdemeyer:

Replying to kcrisman:

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind.

That was for supporting "new" systems for which there was previously no support. Here, we are talking about changing things for a system on which Sage already "passes tests". So going from "passes tests" to "builds" would be a regression.

well, arguably clang support is needed to resurrect the freebsd port, perhaps for dragonbsd support as well. so this is new.

as well, as long as clang is not the default way to build Sage on a system, one should not insist on passing all the tests on this system, right?

[kcrisman]

as well, as long as clang is not the default way to build Sage on a system, one should not insist on passing all the tests on this system, right?

Exactly my thinking. Of course if clang becomes default on OS X that is a very different thing.

[jdemeyer]

Replying to kcrisman:

Of course if clang becomes default on OS X that is a very different thing.

Correct me if I'm wrong, but isn't that part of what this ticket does?

[jdemeyer]

Replying to dimpase:

as long as clang is not the default way to build Sage on a system

Correct me again if I'm wrong, but this ticket does make Clang the default in the sense that GCC won't be installed if Clang is installed.

[jdemeyer]

And if I'm wrong on any of the two points I just mentioned, it would be good if somebody could update the ticket description to explain precisely what this ticket does.

[fbissey]

You are quite correct. In fact this is better/worse than that. Any compiler C/C++ compiler that is properly detected by AC_PROG_CC and AC_PROG_CXX and sets the autoconf variable GCC to yes, can and will be used. It could be limited to clang but I think it is better to open up a bit. Other compilers are not the default on "normal" systems.

So, in the absence of a working gcc/g++, if clang/clang++ is present and working, it will be used. In the case of OS X where /usr/bin/gcc is a wrapper to clang, it will be used. Unless someone overrides it.

Unless the variables CC/CXX are set, gcc will be tested first. Because of the fact on OS X, xcode installs usr/bin/gcc that means changing the default on OS X that's absolutely correct. If you hadn't given any thoughts to it, Jeroen just put your nose in it. I should add that when I took over this ticket it was quite clear to me that it was the objective because gcc keep getting broken in funny ways, which means we do things that end up having unintended consequences. That and the fact that building gcc is probably the package that take the longest to build on OS X.

I make no bones or apologies about the fact that this ticket makes clang the default on OS X.

I am updating the description and title. Also I think the ticket is not "meta" anymore.

The status is doctests pass and all standard packages' testsuite pass except for two (one with an extra ticket). Is it good enough to change the default now? My impression is that the people using OS X are quite happy with the deal but it will trigger buildbot failures unless we do something about eclib's testsuite.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​59910fb

Merge branch 'develop' into clang-progress

[fbissey]

eclib is a tough nut to crack because it appears the precision is not as good as the original design may have expected it to be. Error handling and propagation is a difficult business and the use of clang just uncovered it. In some of tests we have 45 digits of precision instead of the expected 50 for example. Not a big deal for most numerical applications but not good when your tests expect 50 digits to be correct. And increasing precision can cause the sign flip flop when quantities are close to (or actually should be) 0.

[dimpase]

Can we have an update to the current beta?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​632b7bc

Merge branch 'develop' into clang-progress

[fbissey]

Replying to dimpase:

Can we have an update to the current beta?

I guess that's what you wanted.

[dimpase]

We have access to a 24-core OSX server now. This should be useful for this ticket :-)

[dimpase]

with SAGE_INSTALL_GCC=no, I get

configure: error: SAGE_INSTALL_GCC is set to 'no', but a Fortran compiler is missing

without it, gcc spkg gets installed, and then gfortran spkg install fails with "gcc already installed".

[dimpase]

specifically, in configure I get

ac_ext=${ac_fc_srcext-f}
ac_compile='$FC -c $FCFLAGS $ac_fcflags_srcext conftest.$ac_ext >&5'
ac_link='$FC -o conftest$ac_exeext $FCFLAGS $LDFLAGS $ac_fcflags_srcext conftest.$ac_ext $LIBS >&5'
ac_compiler_gnu=$ac_cv_fc_compiler_gnu

if test -z "$FC"; then

    if test x$SAGE_INSTALL_GCC = xexists; then
        true  # Do nothing if already installed
    elif test x$SAGE_INSTALL_GCC = xno; then
        as_fn_error $? "SAGE_INSTALL_GCC is set to 'no', but a Fortran compiler is missing" "$LINENO" 5
    else
        { $as_echo "$as_me:${as_lineno-$LINENO}: Installing GCC because a Fortran compiler is missing" >&5
$as_echo "$as_me: Installing GCC because a Fortran compiler is missing" >&6;}
        need_to_install_gcc=yes
    fi

else
   # see http://www.gnu.org/software/hello/manual/autoconf/Fortran-Compiler.html

but I don't understand how as_fn_error $? "SAGE_INSTALL_GCC is set to 'no', but a Fortran compiler is missing" "$LINENO" 5 is generated from configure.ac. I presume it comes from expansion of

AC_LANG(Fortran)
if test -z "$FC"; then
    need_to_install_gfortran=yes
else
   # see http://www.gnu.org/software/hello/manual/autoconf/Fortran-Compiler.html

but how?

[dimpase]

More frustrating is that even though replacing as_fn_error ... in the previous comment with true allows the build with clang to complete, by doing

$ SAGE_INSTALL_GCC="no" ./configure CC=cc CXX=c++
$ make 

a subsequent, say, ./sage -f r starts a full build of gcc. Arrrgh, that gcc virus. I see

...
env SAGE_BUILD_TOOLCHAIN=yes make toolchain
sage-logger -p 'sage-spkg gcc-7.2.0' '/Users/dimpase/Sage/sagetrac-mirror/logs/pkgs/gcc-7.2.0.log'
[gcc-7.2.0] Found local metadata for gcc-7.2.0
[gcc-7.2.0] Using cached file /Users/dimpase/Sage/sagetrac-mirror/upstream/gcc-7.2.0.tar.xz
...

I never heard of SAGE_BUILD_TOOLCHAIN...

[dimpase]

OK, the latter problem is that the gcc build was still loaded via configure.ac's

AS_ECHO_N(>&7 ['TOOLCHAIN = $(inst_gcc)'])

and this got through into the build system.

---

It looks as if either the TOOLCHAIN business is obsolete and ought to be removed, or it needs to be updated to the case of the use of clang (i.e. it would be 'TOOLCHAIN = $(inst_gfortran)'). (I don't know its purpose anyway, and it does not seem to be documented).

[fbissey]

Sorry to have gone dark on you. I will inspect all that with a fresh mind sometime tomorrow.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8845edb

Merge branch 'develop' into clang-progress

[fbissey]

I checked the TOOLCHAIN business and I believe it is all correct. TOOLCHAIN can be set to $(inst_gcc) only when gcc needs to be built - if you use ccache, I think it is added there too. If it is still set when gcc doesn't need to be built there is definitely a bug somewhere in the configure script. Are you sure you have run ./bootstrap before configuring?

[dimpase]

I did run bootstrap -d. Not sure if the downloaded configure tarball is the correct one. I tried to build autotools our of the tree, but it complains about old m4. And I cannot build m4 - it builds, but crashes with Illegal instruction: 4. Arrgh...

How does one create the configure tarball for downloading?

[fbissey]

Replying to dimpase:

I did run bootstrap -d. Not sure if the downloaded configure tarball is the correct one. I tried to build autotools our of the tree, but it complains about old m4. And I cannot build m4 - it builds, but crashes with Illegal instruction: 4. Arrgh...

How does one create the configure tarball for downloading?

Usually ./bootstrap -s -i. The -i increase the version number. Just autoreconf -i does the job. Not sure about the problem with the autotool spkg.

[jdemeyer]

The git log here is insane. Allow me to squash it...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d3d8317

New configure tarball

[jdemeyer]

Replying to dimpase:

How does one create the configure tarball for downloading?

I just did that for you :-)

[dimpase]

OK, so configure tarball was the culprit. Once I installed autotools out of the tree, while using the system-provided m4, I was able to run ./bootstrap, and build with clang just fine.

Sage's autotools package is broken: it tries building its own m4, but it does not work:

bash-3.2$ ls -l local/bin/m4
-rwxr-xr-x  1 dimpase  staff  293092 Jan  5 14:14 local/bin/m4
bash-3.2$ ./local/bin/m4
Illegal instruction: 4

and then unsurprisingly one gets

...
sed -e 's,@''datadir''@,/Users/dimpase/Sage/sagetrac-mirror/local/autoconf-2.13.rc1/share/autoconf,g' -e 's,@''PERL''@,/usr/bin/perl,g' autoscan.pl > autoscan.tmp && chmod +x autoscan.tmp && mv autoscan.tmp autoscan
Error: Autoconf requires GNU m4 1.1 or later
make[3]: *** [autoconf.m4f] Error 1
make[2]: *** [/Users/dimpase/Sage/sagetrac-mirror/local/autoconf-2.13.rc1] Error 2

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​5b8e8dd	Support downloading of symlinked tarballs
​5ece103	Add a new gfortran spkg and use it when gfortran is not found
​e08f902	New configure tarball

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b4d3067

Mark gfortran as not installed after installing gcc

[jdemeyer]

On Darwin osx 16.7.0 Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64 x86_64:

----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 6444.0 seconds
    cpu time: 33345.3 seconds
    cumulative wall time: 48795.6 seconds

So I think that this ticket is good to go.

[fbissey]

Thanks Jeroen, a few interesting things in there I wouldn't have thought about. So you think it can go in spite of eclib's test suite breaking? Upstream has a fix getting ready for that but I understand changes will be needed sage side to match.

[dimpase]

I've checked that FC=gfortran6 can be provided on FreeBSD.

One more thing for a followup ticket: sort out the case of /bin/sh not being bash (as on FreeBSD) and running with bash as the SHELL.Curently this leads to /bin/sh creeping up in configure, and a breakage.

[jdemeyer]

Replying to fbissey:

So you think it can go in spite of eclib's test suite breaking?

That is really more a question for the release manager. Anyway, it's far more important that the Sage doctests pass.

[jdemeyer]

Replying to dimpase:

One more thing for a followup ticket: sort out the case of /bin/sh not being bash (as on FreeBSD)

See #23448.

[dimpase]

I think it's ready. (I ran more tests on OSX).

[jdemeyer]

I remind you that there is still the dependency #24476 to be reviewed.

[vbraun]

Merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d327bf6

Mark gfortran as not installed after installing gcc

[jdemeyer]

Guessing that the conflict comes from the configure tarball, I just removed that commit.

[vbraun]

merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​c93f7f1	Add a new gfortran spkg and use it when gfortran is not found
​2c2ca7b	Mark gfortran as not installed after installing gcc

[jdemeyer]

Rebased to 8.2.beta3

[vbraun]

Tries to build gfortran on Debian 7 (despite being installed) but fails:

[gfortran-7.2.0] Found local metadata for gfortran-7.2.0
[gfortran-7.2.0] Using cached file /home/buildbot/slave/sage_git/build/upstream/gcc-7.2.0.tar.xz
[gfortran-7.2.0] gfortran-7.2.0
[gfortran-7.2.0] ====================================================
[gfortran-7.2.0] Setting up build directory for gfortran-7.2.0
[gfortran-7.2.0] Finished extraction
[gfortran-7.2.0] No patch files found in ../patches
[gfortran-7.2.0] ****************************************************
[gfortran-7.2.0] Host system:
[gfortran-7.2.0] Linux sagebd07_64s02 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u2~bpo8+1 (2017-01-05) x86_64 GNU/Linux
[gfortran-7.2.0] ****************************************************
[gfortran-7.2.0] C compiler: gcc
[gfortran-7.2.0] C compiler version:
[gfortran-7.2.0] Using built-in specs.
[gfortran-7.2.0] COLLECT_GCC=gcc
[gfortran-7.2.0] COLLECT_LTO_WRAPPER=/home/buildbot/slave/sage_git/build/local/libexec/gcc/x86_64-pc-linux-gnu/7.2.0/lto-wrapper
[gfortran-7.2.0] Target: x86_64-pc-linux-gnu
[gfortran-7.2.0] Configured with: ../src/configure --prefix=/home/buildbot/slave/sage_git/build/local --with-local-prefix=/home/buildbot/slave/sage_git/build/local --with-gmp=/home/buildbot/slave/sage_git/build/local --with-mpfr=/home/buildbot/slave/sage_git/build/local --with-mpc=/home/buildbot/slave/sage_git/build/local --with-system-zlib --disable-multilib --disable-nls --enable-languages=c,c++,fortran --disable-libitm  
[gfortran-7.2.0] Thread model: posix
[gfortran-7.2.0] gcc version 7.2.0 (GCC) 
[gfortran-7.2.0] ****************************************************
[gfortran-7.2.0] Error: gcc is already installed
[gfortran-7.2.0] 
[gfortran-7.2.0] real	0m0.003s
[gfortran-7.2.0] user	0m0.000s
[gfortran-7.2.0] sys	0m0.000s
[gfortran-7.2.0] ************************************************************************
[gfortran-7.2.0] Error installing package gfortran-7.2.0
[gfortran-7.2.0] ************************************************************************

For the record:

buildbot@sagebd07_64s02:~/slave/sage_git/build$ gfortran --version
GNU Fortran (Debian 4.7.2-5) 4.7.2

[dimpase]

Replying to vbraun:

It's not 100% clear what the bot configuration really is. Does it install gcc from Sage, too? If not, what is gcc installed? Does it match gfortran installed, version-vise?

(all this can be found in the output of the main ./configure script.)

[fbissey]

Considering the output, I'd say it is an incremental build. sage's gcc is installed because the OS' one is too old (4.7.2). So there is something in the configure logic that did go wrong and I thought I had guarded and tested against it. At least spkg-install behaved as expected and stopped before causing further dommages.

[dimpase]

Replying to fbissey:

Considering the output, I'd say it is an incremental build. sage's gcc is installed because the OS' one is too old (4.7.2). So there is something in the configure logic that did go wrong and I thought I had guarded and tested against it. At least spkg-install behaved as expected and stopped before causing further dommages.

perhaps just change spkg-install to exit happily if gcc is already installed?

On the other hand it seems that one simply should not do incremental build where this ticket is tested.

[jdemeyer]

Replying to fbissey:

So there is something in the configure logic that did go wrong

I think this is again the typical issue that configure doesn't get updated on the buildbots. The bug is in the release manager workflow, not this ticket. I'll try to provide a workaround.

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

​79c3b7a	Preparations for adding a gfortran package
​ac7816d	Add a new gfortran spkg and use it when gfortran is not found

[vbraun]

Is installed and matches version

buildbot@sagebd07_64s02:~$ gcc --version
gcc (Debian 4.7.2-5) 4.7.2
[...]
buildbot@sagebd07_64s02:~$ gfortran --version
GNU Fortran (Debian 4.7.2-5) 4.7.2

[vbraun]

• The error happend with both incremental and full build
• Only happens on Debian 7, not other distros

[fbissey]

The bit that's supposed to prevent this from happening in configure is

# if GCC needs to be installed GFORTRAN shouldn't be installed
# GCC will provide GFORTRAN in that case
if test $need_to_install_gcc = yes ; then
    need_to_install_gfortran=no
fi

so this test must be failing with the shell used debian 7. It will to be converted into something more universally reliable.

[dimpase]

Could it be that on that Debian 7 system /bin/sh is not bash? This might explain this singularity.

[jdemeyer]

Please read 479 for a possible explanation and see #24579 for a possible fix...

[dimpase]

Why is this in sage-pending?

[fbissey]

Thanks everyone!