Opened 7 years ago

Closed 10 months ago

Last modified 10 months ago

#12426 closed task (fixed)

Make Sage build with clang (3.7+) and make it the default on OS X

Reported by: ohanar Owned by: GeorgSWeber
Priority: major Milestone: sage-8.2
Component: build Keywords: clang porting
Cc: leif, jpflori, dimpase, mkoeppe, isuruf Merged in:
Authors: François Bissey Reviewers: Jeroen Demeyer, John Palmieri, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: ac7816d (Commits) Commit:
Dependencies: #24579 Stopgaps:

Description (last modified by jdemeyer)

Apple stopped supporting GCC in favour of clang. This is to track the status of building Sage with clang with the aim of making it the default compiler used on OS X and other OS where it is the default compiler.


Packages failing their tests


Supporting clang not only on OSX:

  • porting to clang/FreeBSD -- #22679

Attachments (4)

readelf.libgmpxx (1.8 KB) - added by dimpase 2 years ago.
output of readelf on libgmpxx
libc++.readelf (1.6 KB) - added by dimpase 2 years ago.
output of readelf on libc++
gcc-5.4.0.log.gz (175.7 KB) - added by dimpase 19 months ago.
OSX gcc spkg failure
configure-233.tar.gz (95.8 KB) - added by fbissey 16 months ago.

Download all attachments as: .zip

Change History (495)

comment:1 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:2 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:3 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:4 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:5 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:6 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:7 Changed 7 years ago by jdemeyer

I guess many of these tickets are useful for porting in general.

comment:8 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:9 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:10 Changed 7 years ago by ohanar

  • Description modified (diff)

comment:11 Changed 7 years ago by leif

  • Cc leif added

comment:12 Changed 7 years ago by leif

  • Description modified (diff)

comment:13 Changed 7 years ago by leif

  • Description modified (diff)
  • Keywords C++11 porting added

Added further ticket references.

comment:14 Changed 6 years ago by jpflori

  • Cc jpflori added

comment:15 Changed 5 years ago by leif

  • Description modified (diff)
  • Summary changed from make sage build with clang (3.0+) to Make Sage build with clang (3.0+)

Description presumably needs some updating...

comment:16 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:17 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:18 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:19 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:20 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:21 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:22 follow-up: Changed 5 years ago by leif

eclib has been fixed upstream last year (IIRC) as well; currently re-checking whether our current version still builds...

comment:23 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:24 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:25 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:26 Changed 5 years ago by leif

So on Darwin at least, we'd still have to build gfortran... (i.e., GCC with only the Fortran frontend and libgfortran.)

Or resort to f2c again... :-)

comment:27 in reply to: ↑ 22 Changed 5 years ago by leif

  • Description modified (diff)

Replying to leif:

eclib has been fixed upstream last year (IIRC) as well; currently re-checking whether our current version still builds...

(Still) works for me, clang 3.2; make check (and some doctests) passed.

comment:28 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:29 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:30 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:31 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:32 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:33 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:34 Changed 5 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:35 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:36 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:37 Changed 5 years ago by jdemeyer

  • Description modified (diff)

comment:38 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:39 Changed 5 years ago by leif

I think the upcoming GCC 4.9 will help us with Clang as well... ;-)

comment:40 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:41 follow-up: Changed 5 years ago by leif

Checking current status with Sage 6.2 and clang 3.4.1 ...

comment:42 in reply to: ↑ 41 ; follow-up: Changed 5 years ago by leif

Replying to leif:

Checking current status with Sage 6.2 and clang 3.4.1 ...

The following package(s) may have failed to build:

package: atlas-3.10.1.20140210

package: boehm_gc-7.2d.p0

package: elliptic_curves-0.7

package: gf2x-1.1.p0

package: gfan-0.5.p0

package: lcalc-1.23.p12

package: libgap-4.7.4

package: polybori-0.8.3

package: pynac-0.3.2

package: ratpoints-2.1.3.p3

package: symmetrica-2.0.p9

[More to come.]

comment:43 Changed 5 years ago by leif

Goes on with

package: eclib-20140128

package: linbox-1.3.2.p0

package: numpy-1.7.0

package: singular-3.1.6.p1

comment:44 in reply to: ↑ 42 ; follow-up: Changed 5 years ago by leif

Replying to leif:

package: elliptic_curves-0.7

No idea what's going on there (ImportError: No module named _sqlite3); just recompiling sqlite with GCC doesn't help.

comment:45 Changed 5 years ago by leif

Many of the hundreds of C++ errors are of the form

error: friend declaration specifying a default argument must be the only declaration
error: friend declaration specifying a default argument must be a definition

comment:46 in reply to: ↑ 44 Changed 5 years ago by leif

Replying to leif:

Replying to leif:

package: elliptic_curves-0.7

No idea what's going on there (ImportError: No module named _sqlite3); just recompiling sqlite with GCC doesn't help.

Ah, that's a nice one: Python fails to build the _sqlite3 extension because of

clang: error: unknown argument: '-R/path/to/sage-6.2-clang-3.4.1/local/lib'

comment:47 Changed 5 years ago by leif

  • Description modified (diff)

comment:48 Changed 5 years ago by leif

Linbox (without the errors caused by Givaro headers):

../../linbox/solutions/solve.h:314:9: error: no matching conversion for functional-style cast from 'const char [19]' to 'LinBox::LinBoxFailure'

comment:49 Changed 5 years ago by leif

  • Description modified (diff)

NumPy fails to build due to (at least) various problems with xmmintrin.h.

comment:50 Changed 5 years ago by leif

  • Description modified (diff)

comment:51 Changed 5 years ago by leif

  • Description modified (diff)

Ok, removed gf2x and NumPy again; this was my fault, as clang kept including the wrong xmmintrin.h header (i.e., GCC's instead of its own).

comment:52 Changed 5 years ago by leif

  • Description modified (diff)

For Python's distutils, a patch is attached to #16376 in order to not pass -R (but -Wl,-R,...) to clang, such that e.g. the _sqlite3 module also builds with clang.

comment:53 Changed 5 years ago by leif

By the way, R 3.0.2 built for me with clang 3.4.1 without any problems (using gfortran 4.9.0).

comment:54 Changed 5 years ago by leif

  • Description modified (diff)

comment:55 Changed 4 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:56 Changed 4 years ago by aapitzsch

  • Description modified (diff)

comment:57 Changed 4 years ago by leif

  • Milestone changed from sage-6.4 to sage-6.7

comment:58 Changed 4 years ago by leif

SomeoneTM should update the list w.r.t. more recent versions of Sage and clang...

comment:59 Changed 2 years ago by fbissey

  • Description modified (diff)
  • Keywords C++11 removed
  • Milestone changed from sage-6.7 to sage-7.5

I may post a branch here for stuff that I don't feel are in the right shape for their own ticket.

comment:60 Changed 2 years ago by rws

  • Description modified (diff)

comment:61 Changed 2 years ago by fbissey

  • Description modified (diff)

comment:62 Changed 2 years ago by rws

Just to document the problem I had after applying all the patches above. I've given up for now:

[libgap-4.8.3] /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux
/bin/ld: .libs/libgap_la-ariths.o: relocation R_X86_64_32S against `libGAP_ZEROOp
' can not be used when making a shared object; recompile with -fPIC
[libgap-4.8.3] .libs/libgap_la-ariths.o: error adding symbols: Bad value
[libgap-4.8.3] clang-3.7: error: linker command failed with exit code 1 (use -v t
o see invocation)
[libgap-4.8.3] Makefile:503: recipe for target 'libgap.la' failed
[libgap-4.8.3] make[5]: *** [libgap.la] Error 1

OpenSuSE 42.1, clang-3.7, gcc 4.8/5/6 in system, invoked with make distclean; CC=clang CXX=clang++ make

Last edited 2 years ago by rws (previous) (diff)

comment:63 Changed 2 years ago by dimpase

  • Cc dimpase added
  • Summary changed from Make Sage build with clang (3.0+) to Make Sage build with clang (3.7+)

updated the versions we target to more realistic (Xcode 6.4 was based on 3.6.0svn), and meanwhile we are much further, see see https://gist.github.com/yamaya/2924292

comment:64 Changed 2 years ago by fbissey

I am currently working with xcode 8.0 which is clang 3.8.0. fplll-5.0.x compiles but I haven't checked singular 4. No way to check fpylll without sage building first. I'll post some details on the current problems with sagelib tomorrow. That will be interesting :)

Hint stuff outside sagelib may need fixes.

comment:65 Changed 2 years ago by fbissey

So the first failure in building sagelib occurs with sage/finance/time_series.pyx

[sagelib-7.4.rc0] [  1/386] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.o -fno-strict-aliasing
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.c:317:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1777:
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
[sagelib-7.4.rc0] #warning "Using deprecated NumPy API, disable it by " \
[sagelib-7.4.rc0]  ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/finance/time_series.c:15252:3: error: non-void function '__pyx_pf_4sage_7finance_11time_series_10TimeSeries_106numpy' should return a value [-Wreturn-type]
[sagelib-7.4.rc0]   import_array();
[sagelib-7.4.rc0]   ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1532:144: note: expanded from macro 'import_array'
[sagelib-7.4.rc0] #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }
[sagelib-7.4.rc0]                                                                                                                                                ^
[sagelib-7.4.rc0] 1 warning and 1 error generated.

The problem is the import_array statement from numpy, more precisely this bit of code from local/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h

#if PY_VERSION_HEX >= 0x03000000
#define NUMPY_IMPORT_ARRAY_RETVAL NULL
#else
#define NUMPY_IMPORT_ARRAY_RETVAL
#endif

#define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }

As you can see for python < 3 NUMPY_IMPORT_ARRAY_RETVAL is defined but empty, therefore the return statement in import_array is empty. This is completely and mainly rely on gcc to be permissive. NULL is not necessarily a good return value either, since it should be an error code. The whole thing assume or impose something on the part of the function calling import_array that is not true in time_series an I am not sure can be imposed in cython (don't know enough about cython).

A quick and dirty solution is to edit _multiarray.py and make NUMPY_IMPORT_ARRAY_RETVAL NULL always. That get us to the next problem.

comment:66 follow-up: Changed 2 years ago by fbissey

Next failure

[sagelib-7.4.rc0] [  1/373] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.o -fno-strict-aliasing
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1334:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_compute_out_neighborhood_cardinality)(struct __pyx_obj_4sage_6graphs_20graph_decompositions_12fast_digraph_FastDigraph *, int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1335:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_popcount32)(int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] 2 errors generated.

I am not sure what to do with that one. I'll just note line 257 is the one enabled by __GNUC__

#ifndef CYTHON_INLINE
  #if defined(__GNUC__)
    #define CYTHON_INLINE __inline__
  #elif defined(_MSC_VER)
    #define CYTHON_INLINE __inline
  #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L
    #define CYTHON_INLINE inline
  #else
    #define CYTHON_INLINE
  #endif
#endif

So clang pretending to be gcc means that CYTHON_INLINE is defined. Of course clang support the GNU form, but it doesn't support a declaration to be inlined as it is in that code. This is clearly a cython code generation issue.

comment:67 Changed 2 years ago by fbissey

Hand editing the generated cython code allows us to go a bit further until exactly the same error occurs in another file

[sagelib-7.4.rc0] [  1/363] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.o -fno-strict-aliasing
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:1785:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE uint32_t *(*__pyx_f_4sage_6graphs_4base_19static_sparse_graph_has_edge)(__pyx_t_4sage_6graphs_4base_19static_sparse_graph_short_digraph_s *, int, int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:258:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:1786:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE PyObject *(*__pyx_f_4sage_6graphs_4base_19static_sparse_graph_edge_label)(__pyx_t_4sage_6graphs_4base_19static_sparse_graph_short_digraph_s *, uint32_t *); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:258:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/base/static_sparse_backend.c:7758:31: warning: comparison of array '__pyx_v_self->g_rev' not equal to a null pointer is always true [-Wtautological-pointer-compare]
[sagelib-7.4.rc0]   __pyx_t_1 = ((__pyx_v_self->g_rev != NULL) != 0);
[sagelib-7.4.rc0]                 ~~~~~~~~~~~~~~^~~~~    ~~~~
[sagelib-7.4.rc0] 1 warning and 2 errors generated.

That's where I am at right now.

comment:68 Changed 2 years ago by fbissey

And then sage/graphs/graph_decompositions/cutwidth.pyx, sage/libs/mpmath/ext_main.pyx, sage/libs/mpmath/ext_libmp.pyx. Next, my modification of __multiarray_api.h is bitting me.

[sagelib-7.4.rc0] [ 43/291] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.o -fno-strict-aliasing
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.c:309:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/arrayobject.h:4:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:
[sagelib-7.4.rc0] In file included from /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1777:
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: "Using deprecated NumPy API, disable it by "          "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
[sagelib-7.4.rc0] #warning "Using deprecated NumPy API, disable it by " \
[sagelib-7.4.rc0]  ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_double_dense.c:28680:3: error: void function 'initmatrix_double_dense' should not return a value [-Wreturn-type]
[sagelib-7.4.rc0]   import_array();
[sagelib-7.4.rc0]   ^~~~~~~~~~~~~~
[sagelib-7.4.rc0] /Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1532:144: note: expanded from macro 'import_array'
[sagelib-7.4.rc0] #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NUMPY_IMPORT_ARRAY_RETVAL; } }
[sagelib-7.4.rc0]                                                                                                                                                ^      ~~~~~~~~~~~~~~~~~~~~~~~~~
[sagelib-7.4.rc0] 1 warning and 1 error generated.
[sagelib-7.4.rc0] error: command 'clang' failed with exit status 1

May be I'll learn to do the correct thing for the first instance.

comment:69 Changed 2 years ago by fbissey

????????

[sagelib-7.4.rc0] [  4/249] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/matrix/matrix_integer_dense.o -U_LB_DEBUG -DDISABLE_COMMENTATOR -I/Users/fbissey/build/sage/local/include/linbox -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -msse4.1 -mfma -mavx2 -DFFLAS_COMPILED -DFFPACK_COMPILED -I/Users/fbissey/build/sage/local//include -fPIC -I/Users/fbissey/build/sage/local/include -std=gnu++11 -std=c99 -I/Users/fbissey/build/sage/local/include/m4ri -mmmx -msse -msse2 -msse3 -fno-strict-aliasing
[sagelib-7.4.rc0] error: invalid argument '-std=c99' not allowed with 'C++/ObjC++'

Now there's a bug somewhere: -std=gnu++11 -std=c99.

comment:70 Changed 2 years ago by fbissey

Not sure where linbox is pulled from, but that made it .cpp instead of .c. linbox's .pc file brings the -std=gnu++11, module_list.py brings -std=c99. I can remove it from module_list.py but first I'd like to understand where linbox was pulled from.

comment:71 in reply to: ↑ 66 Changed 2 years ago by jdemeyer

Replying to fbissey:

Next failure

[sagelib-7.4.rc0] [  1/373] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p2/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.o -fno-strict-aliasing
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1334:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_compute_out_neighborhood_cardinality)(struct __pyx_obj_4sage_6graphs_20graph_decompositions_12fast_digraph_FastDigraph *, int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:1335:8: error: 'inline' can only appear on functions
[sagelib-7.4.rc0] static CYTHON_INLINE int (*__pyx_f_4sage_6graphs_20graph_decompositions_12fast_digraph_popcount32)(int); /*proto*/
[sagelib-7.4.rc0]        ^
[sagelib-7.4.rc0] /Users/fbissey/build/sage/src/build/cythonized/sage/graphs/graph_decompositions/vertex_separation.c:257:27: note: expanded from macro 'CYTHON_INLINE'
[sagelib-7.4.rc0]     #define CYTHON_INLINE __inline__
[sagelib-7.4.rc0]                           ^
[sagelib-7.4.rc0] 2 errors generated.

That's a genuine problem with the Sage code. GCC only warns for this (which is actually surprising, since the code makes no sense). Ideally, Cython should already refuse this code instead of generating invalid C.

The fix is easy: remove inline from all declarations in .pxd files.

comment:72 Changed 2 years ago by fbissey

Thanks for the pointer Jeroen, that makes two new tickets where I have an idea of what to do. The matrix_integer_dense is a real problem with gcc too

[206/450] x86_64-pc-linux-gnu-g++ -O1 -march=native -pipe -ggdb -fPIC -I/usr/lib64/python2.7/site-packages/cysignals -I/usr/include/openblas -I/usr/include -I/usr/include/python2.7 -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7 -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/sage/ext -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/ext -I/usr/include/python2.7 -c /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/matrix/matrix_integer_dense.cpp -o /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/matrix/matrix_integer_dense.o -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -DOPENBLAS_ARCH_X86_64=1 -DOPENBLAS___64BIT__=1 -O1 -march=native -pipe -ggdb -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -fabi-version=6 -fopenmp -DFFLAS_COMPILED -DFFPACK_COMPILED -O1 -march=native -pipe -ggdb -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mfpmath=sse -fabi-version=6 -I/usr/include/linbox -I/usr/include/openblas -std=c99 -I/usr/include/m4ri -fno-strict-aliasing

Also here it uses g++ rather than gcc. If a .cpp file is generated we should just adjust the options.

comment:73 Changed 2 years ago by fbissey

  • Description modified (diff)

comment:74 follow-up: Changed 2 years ago by fbissey

OK I figured out what is pulled from where now. Lots of declarations directly in matrix_integer_dense.pyx rather than the matching .pxd file. sage/arith/multi_modular.pxd is the only pxd file with a distutils declaration -std=c99 and it is pulled in matrix_integer_dense.pyx by

from sage.arith.multi_modular cimport MultiModularBasis

on line 88. linbox is pulled by

######### linbox interface ##########
from sage.libs.linbox.linbox cimport Linbox_integer_dense
cdef Linbox_integer_dense linbox = Linbox_integer_dense()
USE_LINBOX_POLY = True

line 122 to 125.

In the short term I will forego the distutils declaration in multi_modular.pxd and see what happens combined with some changes to module_list.py.

comment:75 Changed 2 years ago by fbissey

I will open a new ticket about language selection for various pxd and pxi files, I updated the cython inline tickets with new files as I found them. I am now hitting a problem with pynac for which I'll open a new and cc Ralph ASAP.

[sagelib-7.4.rc1] [1/9] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p3/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.o -std=c++11 -fno-strict-aliasing
[sagelib-7.4.rc1] In file included from /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:336:
[sagelib-7.4.rc1] In file included from /Users/fbissey/build/sage/src/sage/symbolic/ginac_wrap.h:11:
[sagelib-7.4.rc1] In file included from /Users/fbissey/build/sage/local/include/pynac/ginac.h:38:
[sagelib-7.4.rc1] /Users/fbissey/build/sage/local/include/pynac/integral.h:44:5: warning: 'GiNaC::integral::evalf' hides overloaded virtual function [-Woverloaded-virtual]
[sagelib-7.4.rc1]         ex evalf(int level=0) const;
[sagelib-7.4.rc1]            ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/local/include/pynac/basic.h:176:13: note: hidden overloaded virtual function 'GiNaC::basic::evalf' declared here: different number of parameters (2 vs 1)
[sagelib-7.4.rc1]         virtual ex evalf(int level = 0, PyObject* parent=nullptr) const;
[sagelib-7.4.rc1]                    ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:27231:30: error: no member named 'operator!=' in 'std::__1::__list_const_iterator<GiNaC::ex, void *>'
[sagelib-7.4.rc1]     __pyx_t_2 = (__pyx_v_itr.operator!=(__pyx_v_lstend) != 0);
[sagelib-7.4.rc1]                  ~~~~~~~~~~~ ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:27446:30: error: no member named 'operator!=' in 'std::__1::__list_const_iterator<GiNaC::ex, void *>'
[sagelib-7.4.rc1]     __pyx_t_3 = (__pyx_v_itr.operator!=(__pyx_v_found.end()) != 0);
[sagelib-7.4.rc1]                  ~~~~~~~~~~~ ^
[sagelib-7.4.rc1] /Users/fbissey/build/sage/src/build/cythonized/sage/symbolic/expression.cpp:28649:30: error: no member named 'operator!=' in 'std::__1::__tree_const_iterator<GiNaC::ex, std::__1::__tree_node<GiNaC::ex, void *> *, long>'
[sagelib-7.4.rc1]     __pyx_t_3 = (__pyx_v_itr.operator!=(__pyx_v_sym_set.end()) != 0);
[sagelib-7.4.rc1]                  ~~~~~~~~~~~ ^
[sagelib-7.4.rc1] 1 warning and 3 errors generated.

Notice that I am down to 9 cythonized file to compile :)

comment:76 Changed 2 years ago by fbissey

  • Description modified (diff)

comment:77 Changed 2 years ago by fbissey

  • Description modified (diff)

comment:78 in reply to: ↑ 74 Changed 2 years ago by jdemeyer

Replying to fbissey:

OK I figured out what is pulled from where now. Lots of declarations directly in matrix_integer_dense.pyx rather than the matching .pxd file. sage/arith/multi_modular.pxd is the only pxd file with a distutils declaration -std=c99

This was a mistake introduced by #17766. The -std=c99 declaration should have been put in the .pyx file, not the .pxd file.

If you make a ticket moving this declaration to multi_modular.pyx, you can set it to positive_review on my behalf.

comment:79 Changed 2 years ago by jdemeyer

comment:80 Changed 2 years ago by jdemeyer

The -std=c99 declaration itself was added in #10281 (yes, I like software archaeology).

comment:81 follow-up: Changed 2 years ago by fbissey

Thanks for pointers Jeroen. What about # distutils: language = ... I also change/added/removed a couple of those. They should also belong to .pyx I presume? That would make things more tidy than the mess I see right now.

comment:82 follow-up: Changed 2 years ago by fbissey

Actually I didn't play as much with language than I remember. I changed only matrix_modn_sparse.pxd from c to c++. I also tried expression.pxd for #21701 but that didn't help - so I won't consider it at this stage. But I think there may a number of .pyx declared c language which become c++ (and .cpp) by inheritance of something in c++ like matrix_integer_dense.pyx.

comment:83 in reply to: ↑ 81 ; follow-up: Changed 2 years ago by jdemeyer

Replying to fbissey:

Thanks for pointers Jeroen. What about # distutils: language = ... I also change/added/removed a couple of those. They should also belong to .pyx I presume?

Not in general. These things have to be considered on a case-by-case basis.

The rule is: the .pxd files should contain the # distutils directives which are needed for the included (cdef extern from ...) files to compile. In src/sage/arith/multi_modular.pxd, there are no included files (it is only Cython code), so there is no reason to add -std=c99. However, it could be that -std=c99 or language = c++ is needed for a particular header file which is used in a .pxd file. In that case, the # distutils directive must be in the .pxd file.

comment:84 in reply to: ↑ 82 ; follow-up: Changed 2 years ago by jdemeyer

Replying to fbissey:

I changed only matrix_modn_sparse.pxd from c to c++.

That is certainly a mistake. There is no reason why it should be c++. I would remove the line # distutils: language = c completely, since I don't see the point of it.

comment:85 follow-up: Changed 2 years ago by jdemeyer

Looking at some # distutils directives, I see many mistakes...

It would indeed be good to clean this up.

comment:86 in reply to: ↑ 84 Changed 2 years ago by fbissey

Replying to jdemeyer:

Replying to fbissey:

I changed only matrix_modn_sparse.pxd from c to c++.

That is certainly a mistake. There is no reason why it should be c++. I would remove the line # distutils: language = c completely, since I don't see the point of it.

OK will try that. I guess a separate ticket for clean up may help slightly with other stuff.

comment:87 in reply to: ↑ 83 Changed 2 years ago by jdemeyer

An example where language = c++ in a .pxd file makes sense is given by Singular: #17254 added # distutils: language = c++ to src/sage/libs/singular/decl.pxd. I think this is correct.

comment:88 in reply to: ↑ 85 Changed 2 years ago by jdemeyer

Replying to jdemeyer:

Looking at some # distutils directives, I see many mistakes...

It would indeed be good to clean this up.

I just created #21749 for this, assuming that you haven't done anything yet.

comment:89 Changed 2 years ago by fbissey

  • Description modified (diff)

comment:90 follow-up: Changed 2 years ago by dimpase

I've been playing with building Sage with clang 3.9 on Linux (gentoo), and it almost works. The only package that fails in a way I can't seem to be able to fix on the spot is pyzmq (see my comments at the end of https://github.com/zeromq/pyzmq/issues/468), and to build sagelib I only need

diff --git a/build/pkgs/libgap/spkg-install b/build/pkgs/libgap/spkg-install
index dd6be37..3a167c9 100755
--- a/build/pkgs/libgap/spkg-install
+++ b/build/pkgs/libgap/spkg-install
@@ -41,6 +41,8 @@ for patch in ../patches/*.patch; do
     fi
 done
 
+CFLAGS="$CFLAGS -fPIC"
+
 echo "Configuring libGAP..."
 ./configure --disable-static \
     CFLAGS="$CFLAGS" CXXFLAGS="$CXXFLAGS" CPPFLAGS="$CPPFLAGS" \
diff --git a/src/sage/libs/m4ri.pxd b/src/sage/libs/m4ri.pxd
index a1fe594..ea07121 100644
--- a/src/sage/libs/m4ri.pxd
+++ b/src/sage/libs/m4ri.pxd
@@ -1,4 +1,3 @@
-# distutils: extra_compile_args = -std=c99
 
 cdef extern from "m4ri/m4ri.h":
     ctypedef int rci_t

After this, I can create a working sage instance by doing ./sage --pip install ipython, which pulls pyzmq-16.0.2, and all gets installed nicely. (there is too much clever AI behind compiler recongnition in Sage's instance of pyzmq - perhaps we should upgrade?)

Running tests now (using ./sage -tp)

Note that pynac fix I proposed on #21701 is not needed here, so it's just an older clang on OSX, it seems.

comment:91 follow-up: Changed 2 years ago by fbissey

Not necessarily. Did you build clang to use clang's version of libstdc++ or are you still using gcc's libstdc++. There is no issue so long as you are using gcc's libstdc++.

comment:92 follow-up: Changed 2 years ago by fbissey

And you need clang-9999 to get the right dependencies to default-libcxx which is llvm's own libstdc++. So I'll assume you are not in the right position to compare with OS X.

comment:93 in reply to: ↑ 91 Changed 2 years ago by dimpase

Replying to fbissey:

Not necessarily. Did you build clang to use clang's version of libstdc++ or are you still using gcc's libstdc++. There is no issue so long as you are using gcc's libstdc++.

hmm, I did emerge --ask clang and it pulled in few other things too.

comment:94 in reply to: ↑ 92 Changed 2 years ago by dimpase

Replying to fbissey:

And you need clang-9999 to get the right dependencies to default-libcxx which is llvm's own libstdc++. So I'll assume you are not in the right position to compare with OS X.

You are right, it still uses gcc's libs. Is there a clean gentoo way to build such a toolchain? All I get is

# emerge --ask clang-9999
!!! 'clang-9999' is not a valid package atom.
!!! Please check ebuild(5) for full details.

PS - OK, I see, it should have been emerge --ask clang-9999...

Last edited 2 years ago by dimpase (previous) (diff)

comment:95 Changed 2 years ago by fbissey

The 9999 denotes a hot version (from git master). There appears to be ebuilds for 3.9.0 (libcxx and libcxxabi) and you should be able to emerge those but you will have to pass some extra arguments to use it instead of gcc's version.

Emerging a 9999 is always a bit tricky. Only do it if you are comfortable with emerge. and for what you did the correct syntax is

emerge =clang-9999

anything including a version number needs "=".

comment:96 in reply to: ↑ 90 Changed 2 years ago by fbissey

Replying to dimpase:

I've been playing with building Sage with clang 3.9 on Linux (gentoo), and it almost works. The only package that fails in a way I can't seem to be able to fix on the spot is pyzmq (see my comments at the end of https://github.com/zeromq/pyzmq/issues/468), and to build sagelib I only need

diff --git a/build/pkgs/libgap/spkg-install b/build/pkgs/libgap/spkg-install
index dd6be37..3a167c9 100755
--- a/build/pkgs/libgap/spkg-install
+++ b/build/pkgs/libgap/spkg-install
@@ -41,6 +41,8 @@ for patch in ../patches/*.patch; do
     fi
 done
 
+CFLAGS="$CFLAGS -fPIC"
+
 echo "Configuring libGAP..."
 ./configure --disable-static \
     CFLAGS="$CFLAGS" CXXFLAGS="$CXXFLAGS" CPPFLAGS="$CPPFLAGS" \
diff --git a/src/sage/libs/m4ri.pxd b/src/sage/libs/m4ri.pxd
index a1fe594..ea07121 100644
--- a/src/sage/libs/m4ri.pxd
+++ b/src/sage/libs/m4ri.pxd
@@ -1,4 +1,3 @@
-# distutils: extra_compile_args = -std=c99
 
 cdef extern from "m4ri/m4ri.h":
     ctypedef int rci_t

After this, I can create a working sage instance by doing ./sage --pip install ipython, which pulls pyzmq-16.0.2, and all gets installed nicely. (there is too much clever AI behind compiler recongnition in Sage's instance of pyzmq - perhaps we should upgrade?)

Running tests now (using ./sage -tp)

Note that pynac fix I proposed on #21701 is not needed here, so it's just an older clang on OSX, it seems.

Hum, Jeroen and I have just added that (m4ri.pxd) and I think it adds a genuine conflict. We will have to examine this carefully.

comment:97 follow-up: Changed 2 years ago by fbissey

OK, was your problem

[sagelib-7.5.beta3] [ 1/59] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p3/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.o -std=c99 -I/Users/fbissey/build/sage/local/include/m4ri -I/Users/fbissey/build/sage/local/include -g -fPIC -Wall -O2 -mmmx -msse -msse2 -msse3 -fno-strict-aliasing
[sagelib-7.5.beta3] error: invalid argument '-std=c99' not allowed with 'C++/ObjC++'

Dima?

comment:98 Changed 2 years ago by fbissey

It may actually be a case of overlinking m4ri is called in more place than it should. sage-on-gentoo compiled with --as-needed only report the following users of m4ri

readelf -d `find /usr/lib64/python2.7/site-packages/sage -name \*.so` | grep -C 6 libm4ri-
 0x0000000000000000 (NULL)               0x0

File: /usr/lib64/python2.7/site-packages/sage/modules/vector_mod2_dense.so

Dynamic section at offset 0x15cd0 contains 27 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x3e20
 0x000000000000000d (FINI)               0x12734
 0x0000000000000019 (INIT_ARRAY)         0x215cb8
--

Dynamic section at offset 0x10b870 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libbrial.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libbrial_groebner.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x21c18
--

File: /usr/lib64/python2.7/site-packages/sage/matrix/matrix_mod2_dense.so

Dynamic section at offset 0x38c60 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libgd.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x6ea8
 0x000000000000000d (FINI)               0x32dc0
 0x0000000000000019 (INIT_ARRAY)         0x238c48
--

File: /usr/lib64/python2.7/site-packages/sage/matrix/matrix_gf2e_dense.so

Dynamic section at offset 0x2ec50 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libm4rie-0.0.20150908.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm4ri-0.0.20140914.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x6400
 0x000000000000000d (FINI)               0x28150
 0x0000000000000019 (INIT_ARRAY)         0x22ec38

Only 4 in total. And sage/rings/polynomial/polynomial_gf2x.so is not one of them.

readelf -d /usr/lib64/python2.7/site-packages/sage/rings/polynomial/polynomial_gf2x.so

Dynamic section at offset 0x38c10 contains 28 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libntl.so.31]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libpython2.7.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x7510

comment:99 Changed 2 years ago by fbissey

Arghhh! sage/rings/polynomial/polynomial_gf2x.pyx uses 2 function from m4ri and both of them are macros, which is why libm4ri doesn't appear in the final link.

comment:100 Changed 2 years ago by fbissey

It seems that all the users of m4ri currently have an entry with appropriate flags in module_list.py so removing the line from m4ri.pxd works. But this has to be sorted in a better way.

comment:101 Changed 2 years ago by fbissey

We have compile! Documentation building in progress. It may take a little bit of time before I can run doctests but we are at a new stage.

comment:102 in reply to: ↑ 97 Changed 2 years ago by dimpase

Replying to fbissey:

OK, was your problem

[sagelib-7.5.beta3] [ 1/59] clang -fno-strict-aliasing -I/Users/fbissey/build/sage/local/var/tmp/sage/build/python2-2.7.10.p3/include -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/cysignals -I/Users/fbissey/build/sage/local/include -I/Users/fbissey/build/sage/local/include/python2.7 -I/Users/fbissey/build/sage/local/lib/python2.7/site-packages/numpy/core/include -I/Users/fbissey/build/sage/src -I/Users/fbissey/build/sage/src/sage/ext -I/Users/fbissey/build/sage/src/build/cythonized -I/Users/fbissey/build/sage/src/build/cythonized/sage/ext -I/Users/fbissey/build/sage/local/include/python2.7 -c /Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.cpp -o build/temp.macosx-10.9-x86_64-2.7/Users/fbissey/build/sage/src/build/cythonized/sage/rings/polynomial/polynomial_gf2x.o -std=c99 -I/Users/fbissey/build/sage/local/include/m4ri -I/Users/fbissey/build/sage/local/include -g -fPIC -Wall -O2 -mmmx -msse -msse2 -msse3 -fno-strict-aliasing
[sagelib-7.5.beta3] error: invalid argument '-std=c99' not allowed with 'C++/ObjC++'

right, this one.

comment:103 follow-up: Changed 2 years ago by jdemeyer

Something like https://github.com/cython/cython/pull/466 could help to better organize compiler flags, but upstream Cython doesn't seem to like it.

comment:104 Changed 2 years ago by dimpase

With a tip by Min, I am also able to complete the Linux/clang (4.0.0) build.

(pyzmq still would need a fix---hopefully upstream...)

comment:105 Changed 2 years ago by dimpase

the results of make ptest are

sage -t src/sage/schemes/elliptic_curves/ell_number_field.py  # 16 doctests failed
sage -t src/sage/plot/graphics.py  # 1 doctest failed
sage -t src/sage/plot/plot.py  # 1 doctest failed
sage -t src/sage/plot/plot3d/shapes2.py  # 1 doctest failed
sage -t src/sage/tests/cmdline.py  # 2 doctests failed
sage -t src/sage/libs/gap/all_documented_functions.py  # Timed out
sage -t src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t src/sage/libs/gap/assigned_names.py  # Timed out
sage -t src/sage/plot/plot3d/base.pyx  # 4 doctests failed
sage -t src/sage/schemes/elliptic_curves/heegner.py  # 37 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_rational_field.py  # 50 doctests failed
sage -t src/sage/tests/book_stein_ent.py  # 3 doctests failed
sage -t src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t src/sage/plot/plot3d/platonic.py  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/padics.py  # 28 doctests failed
sage -t src/sage/schemes/elliptic_curves/sha_tate.py  # 26 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_generic.py  # 14 doctests failed
sage -t src/sage/schemes/elliptic_curves/BSD.py  # 21 doctests failed
sage -t src/sage/schemes/elliptic_curves/ell_point.py  # 13 doctests failed
sage -t src/sage/schemes/elliptic_curves/height.py  # 2 doctests failed
sage -t src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 11 doctests failed
sage -t src/sage/libs/eclib/interface.py  # 34 doctests failed
sage -t src/sage/schemes/elliptic_curves/padic_lseries.py  # 2 doctests failed
sage -t src/sage/libs/eclib/mwrank.pyx  # 3 doctests failed
sage -t src/sage/homology/simplicial_complex.py  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t src/sage/misc/benchmark.py  # 3 doctests failed
sage -t src/sage/tests/benchmark.py  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/ell_tate_curve.py  # 4 doctests failed
sage -t src/sage/libs/lcalc/lcalc_Lfunction.pyx  # 1 doctest failed
sage -t src/sage/schemes/elliptic_curves/ell_egros.py  # 4 doctests failed
sage -t src/sage/interfaces/mwrank.py  # 5 doctests failed
sage -t src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t src/sage/rings/padics/CR_template.pxi  # 2 doctests failed
sage -t src/sage/schemes/elliptic_curves/constructor.py  # 4 doctests failed
sage -t src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
----------------------------------------------------------------------
Total time for all tests: 2376.2 seconds
    cpu time: 6923.9 seconds
    cumulative wall time: 17065.3 seconds

looks like main issues are with number theory things like elliptic curves.

comment:106 Changed 2 years ago by fbissey

Well this is mine on OS X

sage -t --long src/sage/algebras/group_algebra.py  # 5 doctests failed
sage -t --long src/sage/algebras/quaternion_algebra_element.py  # 4 doctests failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra.py  # 8 doctests failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra_element.pyx  # 38 doctests failed
sage -t --long src/sage/arith/misc.py  # 20 doctests failed
sage -t --long src/sage/calculus/calculus.py  # 19 doctests failed
sage -t --long src/sage/categories/quotient_fields.py  # 2 doctests failed
sage -t --long src/sage/categories/rings.py  # 8 doctests failed
sage -t --long src/sage/crypto/lattice.py  # 1 doctest failed
sage -t --long src/sage/crypto/mq/sr.py  # 7 doctests failed
sage -t --long src/sage/geometry/polyhedron/palp_database.py  # 1 doctest failed
sage -t --long src/sage/interfaces/singular.py  # 1 doctest failed
sage -t --long src/sage/libs/pynac/pynac.pxd  # 1 doctest failed
sage -t --long src/sage/matrix/matrix2.pyx  # 3 doctests failed
sage -t --long src/sage/matrix/matrix_integer_dense.pyx  # 13 doctests failed
sage -t --long src/sage/matrix/matrix_rational_dense.pyx  # 1 doctest failed
sage -t --long src/sage/misc/functional.py  # 1 doctest failed
sage -t --long src/sage/modular/etaproducts.py  # 10 doctests failed
sage -t --long src/sage/modules/free_module_integer.py  # 57 doctests failed
sage -t --long src/sage/modules/vector_space_morphism.py  # 3 doctests failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/plot/plot3d/base.pyx  # 4 doctests failed
sage -t --long src/sage/plot/plot3d/platonic.py  # 1 doctest failed
sage -t --long src/sage/plot/plot3d/shapes2.py  # 1 doctest failed
sage -t --long src/sage/repl/preparse.py  # 1 doctest failed
sage -t --long src/sage/rings/complex_number.pyx  # 4 doctests failed
sage -t --long src/sage/rings/integer_ring.pyx  # 5 doctests failed
sage -t --long src/sage/rings/misc.py  # 1 doctest failed
sage -t --long src/sage/rings/real_mpfi.pyx  # 3 doctests failed
sage -t --long src/sage/rings/real_mpfr.pyx  # 1 doctest failed
sage -t --long src/sage/rings/ring.pyx  # 1 doctest failed
sage -t --long src/sage/rings/tests.py  # 2 doctests failed
sage -t --long src/sage/rings/finite_rings/residue_field.pyx  # 14 doctests failed
sage -t --long src/sage/rings/number_field/galois_group.py  # 8 doctests failed
sage -t --long src/sage/rings/number_field/number_field_base.pyx  # 15 doctests failed
sage -t --long src/sage/rings/number_field/number_field_rel.py  # 10 doctests failed
sage -t --long src/sage/rings/number_field/order.py  # 3 doctests failed
sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage -t --long src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long src/sage/rings/polynomial/polynomial_element.pyx  # 7 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx  # 6 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_number_field.pyx  # 19 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/constructor.py  # 11 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/ell_egros.py  # 9 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/ell_rational_field.py  # 15 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/heegner.py  # 33 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/schemes/generic/algebraic_scheme.py  # 10 doctests failed
sage -t --long src/sage/schemes/projective/projective_morphism.py  # 16 doctests failed
sage -t --long src/sage/schemes/toric/divisor.py  # 2 doctests failed
sage -t --long src/sage/schemes/toric/ideal.py  # 45 doctests failed
sage -t --long src/sage/schemes/toric/points.py  # 1 doctest failed
sage -t --long src/sage/schemes/toric/variety.py  # 4 doctests failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/structure/element.pyx  # 5 doctests failed
sage -t --long src/sage/structure/formal_sum.py  # 1 doctest failed
sage -t --long src/sage/tests/french_book/calculus_doctest.py  # 3 doctests failed
sage -t --long src/sage/tests/french_book/domaines_doctest.py  # 3 doctests failed
sage -t --long src/sage/tests/french_book/polynomes.py  # 1 doctest failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/doc/en/prep/Quickstarts/Abstract-Algebra.rst  # 3 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 3 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/nf_introduction.rst  # 5 doctests failed

There is one segfault related to brial to investigate. A lot of plotting fail a specific comparison and there is a strange problem with fplll

        from fpylll import LLL, IntegerMatrix
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/__init__.py", line 6, in <module>
        from .fplll.enumeration import Enumeration, EnumerationError
    ImportError: dlopen(/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so, 2): Symbol not found: __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd
      Referenced from: /Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so
      Expected in: flat namespace
     in /Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so

I am guessing the symbol is not in the "flat namespace". That warning happens a lot in plots:

sage -t --long src/sage/plot/plot.py
**********************************************************************
File "src/sage/plot/plot.py", line 1678, in sage.plot.plot.plot
Failed example:
    plot(2*x+1,(x,0,5),ticks=[[0,1,e,pi,sqrt(20)],2],tick_formatter="latex")
Expected:
    Graphics object consisting of 1 graphics primitive
Got:
    doctest:warning
      File "/Users/fbissey/build/sage-clang/src/bin/sage-runtests", line 89, in <module>
        err = DC.run()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/control.py", line 1130, in run
        self.run_doctests()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/control.py", line 854, in run_doctests
        self.dispatcher.dispatch()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1700, in dispatch
        self.parallel_dispatch()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1590, in parallel_dispatch
        w.start()  # This might take some time
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1866, in start
        super(DocTestWorker, self).start()
      File "/Users/fbissey/build/sage-clang/local/lib/python/multiprocessing/process.py", line 130, in start
        self._popen = Popen(self)
      File "/Users/fbissey/build/sage-clang/local/lib/python/multiprocessing/forking.py", line 126, in __init__
        code = process_obj._bootstrap()
      File "/Users/fbissey/build/sage-clang/local/lib/python/multiprocessing/process.py", line 258, in _bootstrap
        self.run()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1839, in run
        task(self.options, self.outtmpfile, msgpipe, self.result_queue)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2132, in __call__
        runner.run(test)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 636, in run
        return self._run(test, compileflags, out)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.plot.plot.plot[99]>", line 1, in <module>
        plot(Integer(2)*x+Integer(1),(x,Integer(0),Integer(5)),ticks=[[Integer(0),Integer(1),e,pi,sqrt(Integer(20))],Integer(2)],tick_formatter="latex")
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 765, in displayhook
        plain_text, rich_output = self._rich_output_formatter(obj, dict())
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 623, in _rich_output_formatter
        rich_output = self._call_rich_repr(obj, rich_repr_kwds)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 583, in _call_rich_repr
        return obj._rich_repr_(self)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/plot/graphics.py", line 908, in _rich_repr_
        self.save, kwds, file_ext, output_container)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/repl/rich_output/display_manager.py", line 711, in graphics_from_save
        save_function(filename, **kwds)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/misc/decorators.py", line 473, in wrapper
        return func(*args, **kwds)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/sage/plot/graphics.py", line 3206, in save
        figure.tight_layout()
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/figure.py", line 1754, in tight_layout
        rect=rect)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/tight_layout.py", line 349, in get_tight_layout_figure
        pad=pad, h_pad=h_pad, w_pad=w_pad)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/tight_layout.py", line 126, in auto_adjust_subplotpars
        tight_bbox_raw = union([ax.get_tightbbox(renderer) for ax in subplots])
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axes/_base.py", line 3679, in get_tightbbox
        bb_xaxis = self.xaxis.get_tightbbox(renderer)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1073, in get_tightbbox
        ticks_to_draw = self._update_ticks(renderer)
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1038, in _update_ticks
        if not mtransforms.interval_contains(interval_expanded, loc):
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/transforms.py", line 2780, in interval_contains
        ((a < b) and (a <= val and b >= val))
    :
    RuntimeWarning: invalid value encountered in greater_equal
    Graphics object consisting of 1 graphics primitive

comment:107 Changed 2 years ago by fbissey

And the segfaults

  • sage/rings/polynomial/multi_polynomial_sequence.py
    sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 1089 ##
    0
    sage: B.<a,b,c,d> = BooleanPolynomialRing() ## line 1145 ##
    sage: F = Sequence([c + d + b + 1, a + c + d, a*b + c, b*c*d + c]) ## line 1146 ##
    sage: F.eliminate_linear_variables() # everything vanishes ## line 1147 ##
    ------------------------------------------------------------------------
    0   signals.so                          0x000000010604db21 print_backtrace + 65
    1   signals.so                          0x000000010605115c sigdie + 60
    2   signals.so                          0x00000001060510cf cysigs_signal_handler + 351
    3   libsystem_platform.dylib            0x00007fffbdabcbba _sigtramp + 26
    4   ???                                 0x0000000105a5ee00 0x0 + 4389727744
    5   pbori.so                            0x000000019198d45f _ZN8polybori8groebner11linalg_stepERNSt3__16vectorINS_15BoolePolynomialENS1_9allocatorIS3_EEEENS_8BooleSetES8_bbPKc + 479
    6   pbori.so                            0x000000019198cf43 _ZN8polybori8groebner14gauss_on_polysERKNSt3__16vectorINS_15BoolePolynomialENS1_9allocatorIS3_EEEE + 275
    7   pbori.so                            0x000000019198cc50 _ZL58__pyx_pw_4sage_5rings_10polynomial_5pbori_39gauss_on_polysP7_objectS0_ + 192
    8   libpython2.7.dylib                  0x000000010591714a PyEval_EvalFrameEx + 18394
    9   libpython2.7.dylib                  0x000000010591a3e2 fast_function + 338
    
  • sage/rings/polynomial/pbori.pyx (says abort but segfault a lot before aborting)
    sage: sig_on_count() # check sig_on/off pairings (virtual doctest) ## line 4670 ##
    0
    sage: B.<x0,x1,x2,x3> = BooleanPolynomialRing(4) ## line 4689 ##
    sage: I = B.ideal((x0 + x1 + x2 + x3, x0*x1 + x1*x2 + x0*x3 + x2*x3, x0*x1*x2 + x0*x1*x3 + x0*x2*x3 + x1*x2*x3, x0*x1*x2*x3 + 1)) ## line 4690 ##
    sage: gb = I.groebner_basis() ## line 4694 ##
    sage: f,g,h,i = I.gens() ## line 4695 ##
    sage: f.reduce(gb) ## line 4696 ##
    sage: p = f*g + x0*h + x2*i ## line 4698 ##
    sage: p.reduce(gb) ## line 4699 ##
    sage: p.reduce(I) ## line 4701 ##
    Assertion failed: (result != map.end()), function get, file ../../groebner/include/polybori/groebner/PolyEntryIndices.h, line 115.
    ------------------------------------------------------------------------
    0   signals.so                          0x0000000102c38b21 print_backtrace + 65
    1   signals.so                          0x0000000102c3c15c sigdie + 60
    2   signals.so                          0x0000000102c3c0ad cysigs_signal_handler + 317
    3   libsystem_platform.dylib            0x00007fffbdabcbba _sigtramp + 26
    4   ???                                 0x00007fb02d196af0 0x0 + 140394647612144
    5   libsystem_c.dylib                   0x00007fffbd943420 abort + 129
    6   libsystem_c.dylib                   0x00007fffbd90a893 basename_r + 0
    7   libbrial_groebner.0.dylib           0x000000018e7df737 _ZNK8polybori8groebner16PolyEntryIndices3getINSt3__13mapINS_13BooleExponentEiNS_13MapComparatorIS5_EENS3_9allocatorINS3_4pairIKS5_iEEEEEES5_EEiRKT_RKT0_NS1_7uncheckE + 551
    8   libbrial_groebner.0.dylib           0x000000018e7df4ff _ZNK8polybori8groebner16PolyEntryIndices3getINS1_7uncheckEEEiRKNS_13BooleExponentET_ + 47
    9   libbrial_groebner.0.dylib           0x000000018e7df4bd _ZNK8polybori8groebner16PolyEntryIndicesclINS_13BooleExponentEEEiRKT_ + 29
    10  libbrial_groebner.0.dylib           0x000000018e79805c _ZNK8polybori8groebner15PolyEntryVector5indexINS_13BooleExponentEEEmRKT_ + 44
    11  libbrial_groebner.0.dylib           0x000000018e7e79ad _ZN8polybori8groebner15PolyEntryVectorclINS_13BooleExponentEEENS0_18PolyEntryReferenceERKT_ + 61
    12  libbrial_groebner.0.dylib           0x000000018e819ea7 _ZNK8polybori8groebner20SetAssociatedMinimalINS_13BooleExponentELb0EEclERKS2_ + 39
    13  libbrial_groebner.0.dylib           0x000000018e81808a _ZN8polybori8groebner17ReductionStrategy28unmarkNonMinimalLeadingTermsENS_8BooleSetE + 186
    14  libbrial_groebner.0.dylib           0x000000018e817e16 _ZN8polybori8groebner17ReductionStrategy19setupSetsForElementERKNS0_9PolyEntryE + 262
    15  pbori.so                            0x000000018e5410c1 _ZL76__pyx_pw_4sage_5rings_10polynomial_5pbori_17ReductionStrategy_5add_generatorP7_objectS0_ + 177
    16  pbori.so                            0x000000018e4cdc9a _ZL25__Pyx_PyObject_CallOneArgP7_objectS0_ + 154
    17  pbori.so                            0x000000018e529621 _ZL71__pyx_pw_4sage_5rings_10polynomial_5pbori_17BooleanPolynomial_131reduceP7_objectS0_ + 2289
    18  libpython2.7.dylib                  0x000000010250614a PyEval_EvalFrameEx + 18394
    

comment:108 in reply to: ↑ 103 Changed 2 years ago by fbissey

Replying to jdemeyer:

Something like https://github.com/cython/cython/pull/466 could help to better organize compiler flags, but upstream Cython doesn't seem to like it.

Yes it would help. I am a bit unwilling to ship a change of that kind if upstream is unwilling. They don't seem to be interested. In the meantime what should we do with that line in pynac.pxd? If we don't have that functionality in cython it will have to go for now.

I also had found a way to get -std=c99 included in CFLAGS provided by m4ri.pc upstream but after that experience, I am not so sure I want it added there anymore. That's asking for trouble when you mix languages. You'd literally need a break down in CPPFLAGS, CFLAGS, CXXFLAGS and so on, to avoid accidents.

comment:109 Changed 2 years ago by dimpase

I've installed libc++, and while rebuilding from scratch using it, hit a bug in linbox: (in linbox/field/gf2.h)

#if !defined(__PATHCC__) && !(defined(__APPLE__) && defined(__clang__))
#define stdBitReference std::_Bit_reference
#else
#define stdBitReference std::vector<bool>::reference
#endif

that appears to assume that clang not on OSX much be using libstdc++.

By the way, I am still using low-level gcc ABI, e.g. this is a typical ldd output on a C++ library (notice libgcc_s.so there):

$ ldd local/lib/libgmpxx.so
	linux-vdso.so.1 (0x00007ffc1a59f000)
	libgmp.so.16 => /home/dima/Sage/sage-tree/local/lib/libgmp.so.16 (0x00007f5ef4323000)
	libc++.so.1 => /usr/lib64/libc++.so.1 (0x00007f5ef403c000)
	libcxxrt.so.1 => /usr/lib64/libcxxrt.so.1 (0x00007f5ef3e18000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/5.4.0/libgcc_s.so.1 (0x00007f5ef3c01000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f5ef3904000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f5ef356a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5ef334e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f5ef3149000)
	/lib64/ld-linux-x86-64.so.2 (0x00005610d175f000)

comment:110 follow-up: Changed 2 years ago by fbissey

Interesting. I cannot comment on why clang doesn't provide its own libgcc_s or equivalent. Hum, output of readelf -d local/lib/libgmpxx.so and ldd -r /usr/lib64/libc++.so.1 please!

As for linbox I think this is ridiculous. libstdc++ and libc++ are supposed to be feature complete to C++11. Surely you can use the same declaration with both. Issue to open with linbox in my opinion.

comment:111 in reply to: ↑ 110 Changed 2 years ago by dimpase

Replying to fbissey:

Interesting. I cannot comment on why clang doesn't provide its own libgcc_s or equivalent. Hum, output of readelf -d local/lib/libgmpxx.so and ldd -r /usr/lib64/libc++.so.1 please!

What is the replacement of libgcc_s? compiler-rt or libcxxabi? It is probably due to me using clang-3.9 rather than clang-9999 that it uses libgcc_s.

As for linbox I think this is ridiculous. libstdc++ and libc++ are supposed to be feature complete to C++11. Surely you can use the same declaration with both. Issue to open with linbox in my opinion.

OK, I've opened #21977.

Changed 2 years ago by dimpase

output of readelf on libgmpxx

Changed 2 years ago by dimpase

output of readelf on libc++

comment:112 follow-up: Changed 2 years ago by dimpase

the next problem is in building sagelib, and there things are totally misconfigured: matrix_modn_dense_float.cpp is compiled with clang, not with clang++ (and with option -std=gnu++11 for some reason) I get an error saying, most probably, that it does not expect C++ here...

[sagelib-7.5.beta3] [  1/237] clang -fno-strict-aliasing -OPT:Olimit=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/dima/Sage/sage-tree/local/lib/python2.7/site-packages/cysignals -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local/include/python2.7 -I/home/dima/Sage/sage-tree/local/lib/python2.7/site-packages/numpy/core/include -I/home/dima/Sage/sage-tree/src -I/home/dima/Sage/sage-tree/src/sage/ext -I/home/dima/Sage/sage-tree/src/build/cythonized -I/home/dima/Sage/sage-tree/src/build/cythonized/sage/ext -I/home/dima/Sage/sage-tree/local/include/python2.7 -c /home/dima/Sage/sage-tree/src/build/cythonized/sage/matrix/matrix_modn_dense_float.cpp -o build/temp.linux-x86_64-2.7/home/dima/Sage/sage-tree/src/build/cythonized/sage/matrix/matrix_modn_dense_float.o -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -fPIC -std=gnu++11 -msse4.1 -mfma -mavx2 -DFFLAS_COMPILED -DFFPACK_COMPILED -fPIC -std=gnu++11 -I/home/dima/Sage/sage-tree/local/include/linbox -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local//include -I/home/dima/Sage/sage-tree/local/include -fPIC -std=gnu++11 -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local//include -I/home/dima/Sage/sage-tree/local/include -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -fPIC -std=gnu++11 -msse4.1 -mfma -mavx2 -DFFLAS_COMPILED -DFFPACK_COMPILED -fPIC -std=gnu++11 -I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-tree/local//include -I/home/dima/Sage/sage-tree/local/include -fno-strict-aliasing

I wonder why the clang build of sagelib works on OSX - is it because distutils understand what to do in this case?

comment:113 Changed 2 years ago by fbissey

I occasionally have seen .cpp files compiled with clang usually without trouble. And matrix_modn_dense_float is one such case. What is the error exactly?

comment:114 in reply to: ↑ 112 ; follow-up: Changed 2 years ago by jdemeyer

Replying to dimpase:

the next problem is in building sagelib, and there things are totally misconfigured: matrix_modn_dense_float.cpp is compiled with clang, not with clang++ (and with option -std=gnu++11 for some reason)

This is https://bugs.python.org/issue1222585 (an 11-year old bug!)

Last edited 2 years ago by jdemeyer (previous) (diff)

comment:115 Changed 2 years ago by fbissey

I am familiar with it. You wouldn't believe how broken building python or python package is broken on AIX and that's one of the referred bugs for that. On AIX I have to hand hold the building of matplotlib, numpy and scipy, compiling manually the stuff that fails because of wrong compilers and sometimes wrong compiler flags. The way it is broken feels completely magical.

comment:116 in reply to: ↑ 114 ; follow-up: Changed 2 years ago by dimpase

Replying to jdemeyer:

Replying to dimpase:

the next problem is in building sagelib, and there things are totally misconfigured: matrix_modn_dense_float.cpp is compiled with clang, not with clang++ (and with option -std=gnu++11 for some reason)

This is https://bugs.python.org/issue1222585 (an 11-year old bug!)

they say it is enhancement. Technically speaking, building C++ extensions is just not supported... We should just switch to Julia :-)

Anyhow, I suppose the problem I face is that clang has no idea about STL library implementation it should use (well, clang++ does have an appropriate flag (-stdlib=), clang probably just ignores such a flag). It's just an accident that it works with gcc.

comment:117 follow-up: Changed 2 years ago by fbissey

Can you just post the error, so we can see it?

comment:118 in reply to: ↑ 116 ; follow-up: Changed 2 years ago by jdemeyer

Replying to dimpase:

It's just an accident that it works with gcc.

Not really because gcc looks at the extension of the file to be compiled and decides what to do depending on that. If you compile a .cpp file with gcc, I guess it behaves exactly the same as compiling that file with g++. Doesn't clang do the same?

comment:119 in reply to: ↑ 117 Changed 2 years ago by dimpase

Replying to fbissey:

Can you just post the error, so we can see it?

 clang -fno-strict-aliasing -OPT:Olim
it=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/hom
e/dima/Sage/sage-tree/local/lib/python2.7/site-packages/cysignals 
-I/home/dima/Sage/sage-tree/local/include -I/home/dima/Sage/sage-t
ree/local/include/python2.7 -I/home/dima/Sage/sage-tree/local/lib/
python2.7/site-packages/numpy/core/include -I/home/dima/Sage/sage-
tree/src -I/home/dima/Sage/sage-tree/src/sage/ext -I/home/dima/Sag
e/sage-tree/src/build/cythonized -I/home/dima/Sage/sage-tree/src/b
uild/cythonized/sage/ext -I/home/dima/Sage/sage-tree/local/include
/python2.7 -c /home/dima/Sage/sage-tree/src/build/cythonized/sage/
matroids/lean_matrix.c -o build/temp.linux-x86_64-2.7/home/dima/Sa
ge/sage-tree/src/build/cythonized/sage/matroids/lean_matrix.o -fno
-strict-aliasing
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/src/build/cythonized/sage/matrix/matrix_modn_dense_float.cpp:564
:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/matrix/dense-matrix.h:79:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/matrix/densematrix/blas-matrix.h:42:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/vector/stream.h:872:
[sagelib-7.5.beta3] In file included from /home/dima/Sage/sage-tre
e/local/include/linbox/vector/stream-gf2.h:77:
[sagelib-7.5.beta3] /home/dima/Sage/sage-tree/local/include/linbox
/field/gf2.h:1008:10: fatal error: '__bit_reference' file not foun
d
[sagelib-7.5.beta3] #include <__bit_reference>

__bit_reference is a header file from libc++; which clang++ finds as I invoke it as clang++ -stdlib=libc++. But clang has no idea about this parameter, and thus it cannot find it. It is probably not simply including the corresponding directory with -I parameter, as more things from STL are most probably have to be included by default.

comment:120 in reply to: ↑ 118 Changed 2 years ago by dimpase

Replying to jdemeyer:

Replying to dimpase:

It's just an accident that it works with gcc.

Not really because gcc looks at the extension of the file to be compiled and decides what to do depending on that. If you compile a .cpp file with gcc, I guess it behaves exactly the same as compiling that file with g++. Doesn't clang do the same?

I am not sure. By the way, the file in question, where the error pops up, (local/include/linbox/fields/gf2.h) is a C++ file, but is has extension .h, indicating plain C file, I think. (well, it is included from a .cpp file, so in that sense it is meant to be C++).

Besides, g++ has parameters that gcc does not have, such as -nostdinc++ (needed if you want use an STL implementation different from libstdc++), and so I don't see how this could at all fly with gcc if these g++-only parmeters are to be invoked.

Last edited 2 years ago by dimpase (previous) (diff)

comment:121 Changed 2 years ago by fbissey

Well, gcc and clang are not the real compilers, they are drivers. In the case of gcc the real compiler is cc1 for example. So you can pass a c++ file to compile to gcc, and it may be recognised as c++ and passed to cc1plus but the flags that are usually set by g++ won't be present, only the one set by gcc. In the case of gnu compilers it looks like gcc and g++ have the same include flags.

So for example, while it is possible to produce a .o with gcc from a C++ source (and same with clang if you are lucky) it is impossible to link a C++ program with gcc/clang instead of g++/clang++ (without extra manual flags that is) because only the C++ driver knows to add the right flags for the standard C++ library.

I notice for example that matrix_modn_dense_float.cpp is compiled with clang on OS X but the .so file is linked with clang++.

comment:122 Changed 2 years ago by fbissey

On OS X, all the .cpp files are compiled with clang and then linked with clang++. No exceptions. I cannot check the equivalent on linux until tomorrow but I suspect it is the same.

comment:123 follow-up: Changed 2 years ago by fbissey

The version of python shipped in Gentoo actually includes ​https://bugs.python.org/issue1222585 (and yes that means in sage-on-gentoo g++ is used for compiling in the right places). I would be supportive to include that patch in sage's python for once.

comment:124 in reply to: ↑ 123 ; follow-up: Changed 2 years ago by dimpase

Replying to fbissey:

The version of python shipped in Gentoo actually includes ​https://bugs.python.org/issue1222585 (and yes that means in sage-on-gentoo g++ is used for compiling in the right places). I would be supportive to include that patch in sage's python for once.

How does one find out what gentoo is including here?

comment:125 in reply to: ↑ 124 Changed 2 years ago by fbissey

Replying to dimpase:

Replying to fbissey:

The version of python shipped in Gentoo actually includes ​https://bugs.python.org/issue1222585 (and yes that means in sage-on-gentoo g++ is used for compiling in the right places). I would be supportive to include that patch in sage's python for once.

How does one find out what gentoo is including here?

I am assuming you want to brew your own :) In your DISTFILES folder (by default /usr/portage/distfiles) there should be a python-gentoo-patches-2.7.10-0.tar.xz tarball (and a 2.7.12-0 one if you upgraded recently) there are a few other things but inside that tarball are all main - unconditional - patches. patches/21_all_distutils_c++.patch inside that tarball is what you are after if you want to test it. It should be identical to what's on the python tracker for python 2.7 since it's the same author....

comment:126 Changed 21 months ago by jdemeyer

  • Description modified (diff)

comment:127 Changed 20 months ago by mkoeppe

  • Description modified (diff)
  • Milestone changed from sage-7.5 to sage-7.6

comment:128 follow-up: Changed 20 months ago by fbissey

I really need another machine than my OS X laptop to work on this. Do we have some resources I could tap remotely?

comment:129 in reply to: ↑ 128 Changed 20 months ago by dimpase

Replying to fbissey:

I really need another machine than my OS X laptop to work on this. Do we have some resources I could tap remotely?

Not sure if anyone has such resources in the open; indeed, it's illegal to run OSX on non-Apple hardware, although the net is full of instructions on how to e.g. hack VMWarePlayer to make it run OSX on non-Apple hardware.

I'm thinking that perhaps resurrecting FreeBSD port, which as of version 10 has clang as the default toolchain, is the best bet for a platform on which port to clang can be completed without extra hassles.

comment:130 Changed 20 months ago by mkoeppe

  • Cc mkoeppe added

comment:131 Changed 20 months ago by fbissey

OS X, sage 7.6.rc2

----------------------------------------------------------------------
sage -t --long src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage -t --long src/sage/plot/plot.py  # Timed out after testing finished
sage -t --long src/sage/schemes/projective/projective_morphism.py  # 16 doctests failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/ell_rational_field.py  # 17 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/heegner.py  # 33 doctests failed
sage -t --long src/sage/arith/misc.py  # 20 doctests failed
sage -t --long src/sage/matrix/matrix2.pyx  # 3 doctests failed
sage -t --long src/sage/rings/tests.py  # 2 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_element.pyx  # 7 doctests failed
sage -t --long src/sage/libs/ppl.pyx  # Timed out after testing finished
sage -t --long src/sage/misc/functional.py  # 1 doctest failed
sage -t --long src/sage/structure/element.pyx  # 5 doctests failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/sage/matrix/matrix_integer_dense.pyx  # 13 doctests failed
sage -t --long src/sage/schemes/generic/algebraic_scheme.py  # 10 doctests failed
sage -t --long src/sage/schemes/toric/divisor.py  # 2 doctests failed
sage -t --long src/sage/schemes/toric/variety.py  # 4 doctests failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra.py  # 8 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/elliptic_curves.rst  # 3 doctests failed
sage -t --long src/sage/calculus/calculus.py  # 19 doctests failed
sage -t --long src/sage/tests/french_book/calculus_doctest.py  # 3 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/crypto/mq/sr.py  # 7 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_zmod_flint.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/constructor.py  # 11 doctests failed
sage -t --long src/sage/rings/real_mpfi.pyx  # 3 doctests failed
sage -t --long src/sage/rings/complex_number.pyx  # 4 doctests failed
sage -t --long src/sage/schemes/toric/points.py  # 1 doctest failed
sage -t --long src/sage/libs/pynac/pynac.pxd  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/ell_egros.py  # 9 doctests failed
sage -t --long src/sage/rings/real_mpfr.pyx  # 1 doctest failed
sage -t --long src/sage/rings/number_field/number_field_rel.py  # 10 doctests failed
sage -t --long src/sage/rings/finite_rings/residue_field.pyx  # 14 doctests failed
sage -t --long src/sage/rings/number_field/order.py  # 3 doctests failed
sage -t --long src/sage/rings/number_field/galois_group.py  # 8 doctests failed
sage -t --long src/sage/modular/etaproducts.py  # 10 doctests failed
sage -t --long src/sage/tests/french_book/polynomes.py  # 1 doctest failed
sage -t --long src/sage/tests/french_book/domaines_doctest.py  # 3 doctests failed
sage -t --long src/doc/en/thematic_tutorials/explicit_methods_in_number_theory/nf_introduction.rst  # 5 doctests failed
sage -t --long src/sage/algebras/group_algebra.py  # 5 doctests failed
sage -t --long src/sage/rings/ring.pyx  # 1 doctest failed
sage -t --long src/doc/en/prep/Quickstarts/Abstract-Algebra.rst  # 3 doctests failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/modules/vector_space_morphism.py  # 3 doctests failed
sage -t --long src/sage/modules/free_module_integer.py  # 57 doctests failed
sage -t --long src/sage/geometry/polyhedron/palp_database.py  # 1 doctest failed
sage -t --long src/sage/algebras/quatalg/quaternion_algebra_element.pyx  # 38 doctests failed
sage -t --long src/sage/rings/integer_ring.pyx  # 5 doctests failed
sage -t --long src/sage/categories/rings.py  # 8 doctests failed
sage -t --long src/sage/categories/quotient_fields.py  # 2 doctests failed
sage -t --long src/sage/rings/polynomial/polynomial_modn_dense_ntl.pyx  # 6 doctests failed
sage -t --long src/sage/matrix/matrix_rational_dense.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/toric/ideal.py  # 45 doctests failed
sage -t --long src/sage/structure/formal_sum.py  # 1 doctest failed
sage -t --long src/sage/repl/preparse.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_number_field.pyx  # 19 doctests failed
sage -t --long src/sage/rings/number_field/number_field_base.pyx  # 15 doctests failed
sage -t --long src/sage/crypto/lattice.py  # 1 doctest failed
sage -t --long src/sage/rings/misc.py  # 1 doctest failed
sage -t --long src/sage/algebras/quaternion_algebra_element.py  # 4 doctests failed
----------------------------------------------------------------------

comment:132 Changed 20 months ago by fbissey

OK so diving down. There are two main errors that spams a lot of doctests. One is in polybori - go figure :( - and the other is in fplll. So for fplll we have

    ImportError: dlopen(/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so, 2): Symbol not found: __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd
      Referenced from: /Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/fpylll/fplll/enumeration.so

Searching inside libfplll.dylib we get the following information

nm local/lib/libfplll.2.dylib | grep EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd
000000000000cc50 t __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd

The leading lowercase "t" means that the symbol is "local". Which is probably why enumeration.so cannot see it. That's clearly a subtle problem in the compilation of fplll.

comment:133 Changed 20 months ago by dimpase

A lot of these are related to elliptic curves, more precisely, to eclib interface with sage. This seems to be the 3rd group of errors.

I'll see the author of fplll in few days here in Oxford, perhaps we can have a look.

comment:134 Changed 20 months ago by isuruf

  • Cc isuruf added

comment:135 follow-up: Changed 20 months ago by fbissey

@ isuruf you wouldn't have something for the polybori issue would you?

comment:136 Changed 20 months ago by fbissey

With the patch from isuruf at #22643 we are way down in doctests failures

sage -t --long src/sage/rings/polynomial/multi_polynomial_sequence.py  # Killed due to segmentation fault
sage -t --long src/sage/rings/polynomial/pbori.pyx  # Killed due to abort
sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/sage/crypto/mq/sr.py  # 7 doctests failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/libs/pynac/pynac.pxd  # 1 doctest failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed

comment:137 in reply to: ↑ 135 ; follow-up: Changed 20 months ago by isuruf

Replying to fbissey:

@ isuruf you wouldn't have something for the polybori issue would you?

nope. I have access to osx only through travis-ci. I'm waiting for https://github.com/cython/cython/pull/466 to be merged.

comment:138 in reply to: ↑ 137 Changed 20 months ago by fbissey

Replying to isuruf:

Replying to fbissey:

@ isuruf you wouldn't have something for the polybori issue would you?

nope. I have access to osx only through travis-ci. I'm waiting for https://github.com/cython/cython/pull/466 to be merged.

OK, I had to ask. Now that fplll is sorted, things are clearer. We have

  • polybori segfaulting,
  • missing symbols in pynac - I will get in touch upstream,
  • issues with matplotlib, although it could be interaction with sage types.
  • similarly an issue in numpy.
  • a tolerance problem in lseries_ell.py, the doctest look at a zero value within 1e-17, but we are only 2e-16 of it.

comment:139 Changed 20 months ago by fbissey

  • Description modified (diff)

comment:140 Changed 20 months ago by dimpase

Let's revive the FreeBSD port. FreeBSD as of version 10 uses clang by default, and is closer to OSX in other ways too, so it would be a good free testing ground for the use of clang with Sage. Preliminary checks show that it's relatively little work. I've created #22679 for this.

comment:141 Changed 20 months ago by fbissey

Using BRiAl git master fix the the segfault, forgeting pynac which is in his own ticket, so we are down to

sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed

matplotlib issue:

...
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1038, in _update_ticks
        if not mtransforms.interval_contains(interval_expanded, loc):
      File "/Users/fbissey/build/sage-clang/local/lib/python2.7/site-packages/matplotlib/transforms.py", line 2780, in interval_contains
        ((a < b) and (a <= val and b >= val))
    :
    RuntimeWarning: invalid value encountered in greater_equal

for all of them.

sage -t --long --warn-long 62.6 src/sage/schemes/elliptic_curves/lseries_ell.py
**********************************************************************
File "src/sage/schemes/elliptic_curves/lseries_ell.py", line 377, in sage.schemes.elliptic_curves.lseries_ell.Lseries_ell.twist_values
Failed example:
    vals  # abs tol 1e-17
Expected:
    [(-11, 1.47824342), (-8, 8.9590946e-18), (-7, 1.85307619), (-4, 2.45138938)]
Got:
    [(-11, 1.47824342), (-8, -1.59574955e-16), (-7, 1.85307619), (-4, 2.45138938)]
Tolerance exceeded in 1 of 8:
    8.9590946e-18 vs -1.59574955e-16, tolerance 2e-16 > 1e-17
**********************************************************************

And

sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed

numpy issue

        numpy.float32('1.5') * x
    :
    RuntimeWarning: invalid value encountered in multiply

both of them.

comment:142 Changed 20 months ago by fbissey

  • Description modified (diff)

Turns out the pynac issue is not a proper issue. cython was trying to run the test with g++ instead of clang++ and the mixing was not working unsurprisingly.

comment:143 Changed 20 months ago by dimpase

  • Description modified (diff)

comment:144 follow-up: Changed 20 months ago by fbissey

I now have a working environment where I can do the work on linux and hope it will match the problems on OS X. Have to figure how not build gcc though, as clang passes itself as gcc 4.2.1 which triggers toolchain building and my sage-env hack is OS X specific.

Inserting a clang identification logic isn't easy. It can be done. Interestingly autoconf people would probably say we are wrong to select versions, we should look for compiler features. Admittedly, that's more difficult than just banning gcc versions, even if it is the right thing.

comment:145 in reply to: ↑ 144 Changed 20 months ago by dimpase

Replying to fbissey:

I now have a working environment where I can do the work on linux and hope it will match the problems on OS X.

For clang++, do you use libc++, as OSX does, rather than libstdc++, as a non-bleeding-edge clang-on-Linux would use the latter... (having clang++/libc++ on Linux looked like a very experimental thing to me; without libc++ some C++-specific thing are rather different)

Have to figure how not build gcc though, as clang passes itself as gcc 4.2.1 which triggers toolchain building and my sage-env hack is OS X specific.

how about SAGE_INSTALL_GCC=no? Is this not enough?

Inserting a clang identification logic isn't easy. It can be done. Interestingly autoconf people would probably say we are wrong to select versions, we should look for compiler features. Admittedly, that's more difficult than just banning gcc versions, even if it is the right thing.

Last edited 20 months ago by dimpase (previous) (diff)

comment:146 Changed 20 months ago by fbissey

I have to figure it out but gentoo stabilised clang-3.9 and you can set it up to use libcxx, which I did. Hope there is no extra flags involved.

Checking on SAGE_INSTALL_GCC=no, didn't really think about that. It should work in that particular case. But I already started to put some logic in configure.ac to figure out we are using clang. On OS X where I didn't have a fortran compiler to start with, it would result in an error - of course because I don't have a fortran compiler I want to build gcc. Just not use gcc and g++.

comment:147 Changed 20 months ago by fbissey

Now I have a full logic that automatically skips toolchain building if using clang. Forbidding older version of clang needs another macro (I already added one) but is doable.

diff --git a/configure.ac b/configure.ac
index 52a9d2c..fa3d1d8 100644
--- a/configure.ac
+++ b/configure.ac
@@ -442,20 +442,30 @@ elif test x$GFC != xyes; then
 else
     # Since need_to_install_gcc is "no", we know that
     # at least C, C++ and Fortran compilers are available.
-    # We also know that all compilers are GCC.
-
+    # We also know that all compilers are GCC-like.
     # Find out the compiler versions:
     AX_GCC_VERSION()
     AX_GXX_VERSION()
-
-    # Add the .0 because Debian/Ubuntu gives version numbers like
-    # 4.6 instead of 4.6.4 (Trac #18885)
-    case "$GXX_VERSION.0" in
-        [[0-3]].*|4.[[0-7]].*)
-            # Install our own GCC if the system-provided one is older than gcc-4.8.
-            SAGE_SHOULD_INSTALL_GCC([you have $CXX version $GXX_VERSION, which is quite old]);;
-    esac
-
+    
+    #Figuring out if we are using clang instead of gcc.
+    AC_LANG_PUSH(C) # we are still set on FORTRAN
+    AX_COMPILER_VENDOR()
+    AC_LANG_POP([C])
+    IS_CLANG=no
+    if test "x$ax_cv_c_compiler_vendor" = xclang
+    then IS_CLANG=yes
+    fi
+    if test "x$IS_CLANG" = "xno"
+    then
+        # for gcc we know we want more recent versions
+        # Add the .0 because Debian/Ubuntu gives version numbers like
+        # 4.6 instead of 4.6.4 (Trac #18885)
+        case "$GXX_VERSION.0" in
+            [[0-3]].*|4.[[0-7]].*)
+                # Install our own GCC if the system-provided one is older than gcc-4.8.
+                SAGE_SHOULD_INSTALL_GCC([you have $CXX version $GXX_VERSION, which is quite old]);;
+        esac
+    fi
     # The following tests check that the version of the compilers
     # are all the same.
     if test "$GCC_VERSION" != "$GXX_VERSION"; then
@@ -497,7 +507,10 @@ else
 
     if test "x$fortran_version_string" = x
     then
-        SAGE_SHOULD_INSTALL_GCC([the Fortran compiler is not the same version as the C compiler])
+        if test "x$IS_CLANG" = "xno"
+        then
+           SAGE_SHOULD_INSTALL_GCC([the Fortran compiler is not the same version as the C compiler])
+        fi
     fi
 fi

and using http://www.gnu.org/software/autoconf-archive/ax_compiler_vendor.html

It assumes that you are using the same vendor for CC and CXX. Mixing those would probably trigger an error in configure unless the gnu compiler you are trying to mix with clang is version 4.2.1 precisely.

On OS X where there is no fortran compiler by default, gcc would still be build and the sage-env hack necessary. But you should be able to bring an external gfortran and skip the building of the toolchain.

comment:148 follow-up: Changed 20 months ago by jdemeyer

You can simplify test "x$IS_CLANG" = "xno" to test "$IS_CLANG" = no or even test $IS_CLANG = no if you know that $IS_CLANG is one word :-)

I think it would be more logical to test AX_COMPILER_VENDOR() earlier, when you are actually testing the C++ compiler.

For Fortran, maybe just drop the version check altogether? If clang works with gfortran, then I would guess that gcc/g++ also work with different versions of gfortran.

Last edited 20 months ago by jdemeyer (previous) (diff)

comment:149 in reply to: ↑ 148 Changed 20 months ago by fbissey

Replying to jdemeyer:

You can simplify test "x$IS_CLANG" = "xno" to test "$IS_CLANG" = no :-)

Yes, it went through several iterations before I figured out that I was trying to check the vendor of the fortran compiler :( [which is unknown because the macro only knows how to do C and C++ compilers].

I think it would be more logical to test AX_COMPILER_VENDOR() earlier, when you are actually testing the C++ compiler.

Yes it would make sense.

For Fortran, maybe just drop the version check altogether? If clang works with gfortran, then I would guess that gcc/g++ also work with different versions of gfortran.

Most likely. I would imagine that if it doesn't it is actually broken.

I am not doing a branch at this stage, I am just gathering material for other people to be able to do it themselves (and give me some feedback like you just did of course).

comment:150 Changed 20 months ago by fbissey

And now for something ridiculous. libgap fails to build with clang-3.9.1 on linux

libtool: link: clang -shared   .libs/libgap_la-ariths.o .libs/libgap_la-c_random.o .libs/libgap_la-gmpints.o .libs/libgap_la-objccoll.o .libs/libgap_la-rational.o .libs/libgap_la-system.o .libs/libgap_la-blister.o .libs/libgap_la-c_type1.o .libs/libgap_la-gvars.o .libs/libgap_la-objcftl.o .libs/libgap_la-read.o .libs/libgap_la-tietze.o .libs/libgap_la-bool.o .libs/libgap_la-cyclotom.o .libs/libgap_la-integer.o .libs/libgap_la-objects.o .libs/libgap_la-records.o .libs/libgap_la-vars.o .libs/libgap_la-calls.o .libs/libgap_la-dt.o .libs/libgap_la-intfuncs.o .libs/libgap_la-objfgelm.o .libs/libgap_la-saveload.o .libs/libgap_la-vec8bit.o .libs/libgap_la-c_filt1.o .libs/libgap_la-dteval.o .libs/libgap_la-intrprtr.o .libs/libgap_la-objpcgel.o .libs/libgap_la-scanner.o .libs/libgap_la-vecffe.o .libs/libgap_la-c_meths1.o .libs/libgap_la-exprs.o .libs/libgap_la-iostream.o .libs/libgap_la-objscoll.o .libs/libgap_la-sctable.o .libs/libgap_la-vecgf2.o .libs/libgap_la-code.o .libs/libgap_la-finfield.o .libs/libgap_la-libgap.o .libs/libgap_la-opers.o .libs/libgap_la-set.o .libs/libgap_la-vector.o .libs/libgap_la-compiler.o .libs/libgap_la-funcs.o .libs/libgap_la-listfunc.o .libs/libgap_la-permutat.o .libs/libgap_la-stats.o .libs/libgap_la-weakptr.o .libs/libgap_la-compstat.o .libs/libgap_la-gap.o .libs/libgap_la-listoper.o .libs/libgap_la-plist.o .libs/libgap_la-streams.o .libs/libgap_la-c_oper1.o .libs/libgap_la-lists.o .libs/libgap_la-precord.o .libs/libgap_la-string.o .libs/libgap_la-costab.o .libs/libgap_la-gasman.o .libs/libgap_la-macfloat.o .libs/libgap_la-range.o .libs/libgap_la-sysfiles.o .libs/libgap_la-pperm.o .libs/libgap_la-trans.o .libs/libgap_la-profile.o    -L/home/fbissey/sandbox/git-fork/sage-clang/local/lib -lgmp -lm  -Wl,--version-script=../src/libgap.map -Wl,-rpath -Wl,/home/fbissey/sandbox/git-fork/sage-clang/local/lib   -Wl,-soname -Wl,libgap.so.4 -o .libs/libgap.so.4.8.6
/usr/bin/x86_64-pc-linux-gnu-ld: .libs/libgap_la-ariths.o: relocation R_X86_64_32S against `libGAP_ZEROOp' can not be used when making a shared object; recompile with -fPIC
.libs/libgap_la-ariths.o: error adding symbols: Bad value
clang-3.9: error: linker command failed with exit code 1 (use -v to see invocation)

As it turns out this bit

CPPFLAGS="$CPPFLAGS"' -DSYS_DEFAULT_PATHS=\"'"$SAGE_LOCAL/gap/latest"'\"'

about SYS_DEFAULT_PATHS is breaking the detection of -fPIC -DPIC by autoconf, so everything is compiled without -fPIC and the linking of the shared object fails.

At this point I think

  1. if it is useful make it a configuration option
  2. I don't any of that in sage-on-gentoo. Is this really necessary? I guess I'll figure the answer to that shortly.

comment:151 follow-up: Changed 20 months ago by fbissey

Oh, I see only used for the testsuite. May be I can move that logic there then.

comment:152 Changed 20 months ago by dimpase

yes, I have seen the need for missing -fPIC with clang too, only I never knew why this was broken...

comment:153 Changed 20 months ago by fbissey

  • Description modified (diff)

Hum, yes. Adding the linbox annoyance #21977 to the list. At least that's easily patchable.

comment:154 follow-up: Changed 20 months ago by fbissey

Well pyzmq fails to build.

    clang -fno-strict-aliasing -OPT:Olimit=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/fbissey/sandbox/git-fork/sage-clang/local/include -Izmq/utils -Izmq/backend/cython -Izmq/devices -c build/temp.linux-x86_64-2.7/scratch/vers.c -o build/temp.linux-x86_64-2.7/scratch/vers.o
    clang build/temp.linux-x86_64-2.7/scratch/vers.o -L/home/fbissey/sandbox/git-fork/sage-clang/local/lib -R/home/fbissey/sandbox/git-fork/sage-clang/local/lib -lzmq -lrt -o build/temp.linux-x86_64-2.7/scratch/vers
    Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory


    error: Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory

clang doesn't know to pass -R directly to the linker. For gcc distutils uses -Wl,-R but recons other compilers don't need to do that. From python2.7/distutils/unixccompiler.py

    def runtime_library_dir_option(self, dir):
        # XXX Hackish, at the very least.  See Python bug #445902:
        # http://sourceforge.net/tracker/index.php
        #   ?func=detail&aid=445902&group_id=5470&atid=105470
        # Linkers on different platforms need different options to
        # specify that directories need to be added to the list of
        # directories searched for dependencies when a dynamic library
        # is sought.  GCC has to be told to pass the -R option through
        # to the linker, whereas other compilers just know this.
        # Other compilers may need something slightly different.  At
        # this time, there's no way to determine this information from
        # the configuration data stored in the Python installation, so
        # we use this hack.
        compiler = os.path.basename(sysconfig.get_config_var("CC"))
        if sys.platform[:6] == "darwin":
            # MacOSX's linker doesn't understand the -R flag at all
            return "-L" + dir
        elif sys.platform[:7] == "freebsd":
            return "-Wl,-rpath=" + dir
        elif sys.platform[:5] == "hp-ux":
            if self._is_gcc(compiler):
                return ["-Wl,+s", "-L" + dir]
            return ["+s", "-L" + dir]
        elif sys.platform[:7] == "irix646" or sys.platform[:6] == "osf1V5":
            return ["-rpath", dir]
        elif self._is_gcc(compiler):
            return "-Wl,-R" + dir
        else:
            return "-R" + dir

_is_gcc basically returns true if gcc or g++ is part of the filename of the compiler. So clang on linux, gets the last option when it would need the same as gcc, or freeBSD. And I would also like people who wrote that code to read ld man page so they drop the way they use -R. It is very much a legacy behavior.

comment:155 Changed 20 months ago by jhpalmieri

  • Description modified (diff)

comment:156 follow-up: Changed 20 months ago by fbissey

Started doctesting, definitely need the new eclib. I get all those backtraces right now.

comment:157 in reply to: ↑ 156 Changed 20 months ago by dimpase

Replying to fbissey:

Started doctesting, definitely need the new eclib. I get all those backtraces right now.

there was a return missing in a function computing double, and clang does not forgive this Matlab-speak

comment:158 in reply to: ↑ 154 Changed 20 months ago by fbissey

Replying to fbissey:

Well pyzmq fails to build.

    clang -fno-strict-aliasing -OPT:Olimit=0 -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/home/fbissey/sandbox/git-fork/sage-clang/local/include -Izmq/utils -Izmq/backend/cython -Izmq/devices -c build/temp.linux-x86_64-2.7/scratch/vers.c -o build/temp.linux-x86_64-2.7/scratch/vers.o
    clang build/temp.linux-x86_64-2.7/scratch/vers.o -L/home/fbissey/sandbox/git-fork/sage-clang/local/lib -R/home/fbissey/sandbox/git-fork/sage-clang/local/lib -lzmq -lrt -o build/temp.linux-x86_64-2.7/scratch/vers
    Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory


    error: Error running version detection script:

    build/temp.linux-x86_64-2.7/scratch/vers: error while loading shared libraries: libzmq.so.4: cannot open shared object file: No such file or directory

clang doesn't know to pass -R directly to the linker. For gcc distutils uses -Wl,-R but recons other compilers don't need to do that. From python2.7/distutils/unixccompiler.py

    def runtime_library_dir_option(self, dir):
        # XXX Hackish, at the very least.  See Python bug #445902:
        # http://sourceforge.net/tracker/index.php
        #   ?func=detail&aid=445902&group_id=5470&atid=105470
        # Linkers on different platforms need different options to
        # specify that directories need to be added to the list of
        # directories searched for dependencies when a dynamic library
        # is sought.  GCC has to be told to pass the -R option through
        # to the linker, whereas other compilers just know this.
        # Other compilers may need something slightly different.  At
        # this time, there's no way to determine this information from
        # the configuration data stored in the Python installation, so
        # we use this hack.
        compiler = os.path.basename(sysconfig.get_config_var("CC"))
        if sys.platform[:6] == "darwin":
            # MacOSX's linker doesn't understand the -R flag at all
            return "-L" + dir
        elif sys.platform[:7] == "freebsd":
            return "-Wl,-rpath=" + dir
        elif sys.platform[:5] == "hp-ux":
            if self._is_gcc(compiler):
                return ["-Wl,+s", "-L" + dir]
            return ["+s", "-L" + dir]
        elif sys.platform[:7] == "irix646" or sys.platform[:6] == "osf1V5":
            return ["-rpath", dir]
        elif self._is_gcc(compiler):
            return "-Wl,-R" + dir
        else:
            return "-R" + dir

_is_gcc basically returns true if gcc or g++ is part of the filename of the compiler. So clang on linux, gets the last option when it would need the same as gcc, or freeBSD. And I would also like people who wrote that code to read ld man page so they drop the way they use -R. It is very much a legacy behavior.

upstream issue: http://bugs.python.org/issue25229

comment:159 in reply to: ↑ 151 Changed 20 months ago by dimpase

Replying to fbissey:

Oh, I see only used for the testsuite. May be I can move that logic there then.

I presume you mean CPPFLAGS from comment 150. Do you have a ticket for this?

comment:160 follow-up: Changed 20 months ago by fbissey

Not yet. I haven't fixed the testsuite so far, but I guess that's not an obstacle to opening a new ticket.

comment:161 in reply to: ↑ 160 Changed 20 months ago by dimpase

Replying to fbissey:

Not yet. I haven't fixed the testsuite so far, but I guess that's not an obstacle to opening a new ticket.

This is now #22784

comment:162 Changed 20 months ago by fbissey

  • Authors set to François Bissey
  • Branch set to u/fbissey/clang-start
  • Commit set to 376ed4fe1cfe33a2d0a6d0e256f064a5a26b2f0f
  • Description modified (diff)

I know I said I wouldn't provide a branch, but all in all we are closer to clang support out of the box than I thought we were last week.

So it is more important that things to start the build are available and open to criticism.


New commits:

28bea10Basic changes to get building of gcc to skip if using clang+gfortran from system and always use clang on OS X
376ed4fAdd missing macro

comment:163 Changed 20 months ago by fbissey

  • Description modified (diff)

comment:164 Changed 20 months ago by fbissey

  • Description modified (diff)

comment:165 Changed 19 months ago by dimpase

can we have the following

+    if [ "$UNAME" = "Darwin" ]; then
+        CXX=clang++

less Darwin-centric? (I'd like to have "Darwin" || "FreeBSD")

Perhaps CC should come from a configure parameter (that has a default)

Then, how about making CPP configurable too? (I imagine that for clang on linux you don't want the default cpp, no?)

comment:166 follow-up: Changed 19 months ago by fbissey

I would have thought that you would bring your fortran compiler on freeBSD. That part is only really activated if you compile gcc, which is only necessary if you don't already have a gfortran install.

Anything more complicated should indeed be the object of a configuration.

For CPP, on linux I have clang-cpp installed, yes there is room for stuff there, especially after the experience with the curl ticket.

comment:167 Changed 19 months ago by jdemeyer

where we will assume there were only installed to provide gfortran

Are you sure that it's impossible to just install gfortran without gcc and g++?

That would be the best solution: don't install gcc and g++ at all if clang is available. Then you would need less special cases in sage-env.

comment:168 Changed 19 months ago by fbissey

I don't think it is possible to build gfortran without building gcc/g++, however gfortran is not bootstrapped it is built with the c or c++ compiler so something may be possible. Also there is the possibility of only installing the fortran bits - provided no other dependencies are needed. I'll look into it.

comment:169 in reply to: ↑ 166 Changed 19 months ago by dimpase

Replying to fbissey:

I would have thought that you would bring your fortran compiler on freeBSD. That part is only really activated if you compile gcc, which is only necessary if you don't already have a gfortran install.

I am not sure if system's gfortran is a good idea on FreeBSD, especially as it needs that ugly /lib/libgcc_s replacement. Perhaps Sage's gfortran will not need it. I'd like to try, as replacing /lib/libgcc_s might be a red flag for many...

By the way, what is the correct way to set up global LDFLAGS for Sage building? (Adding them by hand is ugly...)

comment:170 follow-up: Changed 19 months ago by fbissey

Ok first investigations in gfortran

  1. easiest: install the binary package from https://gcc.gnu.org/wiki/GFortranBinaries#MacOS
  2. I tried to follow http://fortranhelp.blogspot.co.nz/2010/09/building-gfortran-from-source-code-on.html but it sets itself to build the c compiler as well.

Yes libgcc_c is probably why I end up build the c compiler.

Last edited 19 months ago by fbissey (previous) (diff)

comment:171 Changed 19 months ago by fbissey

Number 2 still installed gcc but not g++. It was also much, much faster.

comment:172 in reply to: ↑ 170 ; follow-up: Changed 19 months ago by dimpase

Replying to fbissey:

Yes libgcc_c is probably why I end up build the c compiler.

do you mean libgcc_s ?

comment:173 in reply to: ↑ 172 Changed 19 months ago by fbissey

Replying to dimpase:

Replying to fbissey:

Yes libgcc_c is probably why I end up build the c compiler.

do you mean libgcc_s ?

yes

comment:174 Changed 19 months ago by fbissey

OK after a good night sleep and some thinking I think we can tweak the configuration script and the gcc package to trigger the building of gfortan only under certain condition. At install time we can just suppress the gcc and cpp executables.

It is also much faster. It takes several hours (~3) to bootstrap and build a full gcc/g++/gfortran on my poor laptop. It is now the the single most time consuming item, probably longer than the rest combined. It would be reduced to about half an hour which makes building on that machine much more bearable.

comment:175 Changed 19 months ago by git

  • Commit changed from 376ed4fe1cfe33a2d0a6d0e256f064a5a26b2f0f to 74cb5f7ec3645f62d44649c1767ca3b00c633ec4

Branch pushed to git repo; I updated commit sha1. New commits:

74cb5f7Only build gfortran if clang is CC and gfortran is not found.

comment:176 Changed 19 months ago by fbissey

OK this is a first draft. It is quite generic in that if CC is clang and gfortran is not found the gcc package does a minimal install of gfortran.

In the current setup there is no way to force a full gcc build if you wished too. In fact I removed some OS X specific bits for now until I have checked they are still necessary (I don't believe so but I need experimental proof). So right now the branch would fail to build a full gcc on OS X anyway.

I have leveraged sage-env-config in a minimal way, I am open to alternative suggestion as I put something there that is only ever used by the gcc spkg, which makes it an irritant.

Adding spkg-install.in for gcc could be an interesting option and make it more controllable by configure.

comment:177 Changed 19 months ago by fbissey

Argh.... mpir gets initially build without C++ support when building the toolchain. And I have disabled the rebuilding if you just install gfortran. Back to the drawing board.

comment:178 Changed 19 months ago by git

  • Commit changed from 74cb5f7ec3645f62d44649c1767ca3b00c633ec4 to 2ce0c32ff4fb9085d17d25d058bc539130c77e7e

Branch pushed to git repo; I updated commit sha1. New commits:

2ce0c32Restore unconditional re-building of gmp/mpir/mpfr/mpc

comment:179 Changed 19 months ago by fbissey

Going back one step. That little adventure made me think we need to leverage autoconf more at the spkg-install level but that could be done orthogonally to this ticket.

Last edited 19 months ago by fbissey (previous) (diff)

comment:180 Changed 19 months ago by git

  • Commit changed from 2ce0c32ff4fb9085d17d25d058bc539130c77e7e to fca84601cfe2c9dc0288950c671c53c64bcde411

Branch pushed to git repo; I updated commit sha1. New commits:

fca8460Improve the comments in gcc

comment:181 Changed 19 months ago by fbissey

OK this is fairly good. With this ticket and #22677 sage builds on OS X with clang out of the box and install just gfortran (and a few other bits of gcc along the way). And only the following doctests are failing

sage -t --long src/doc/en/prep/Calculus.rst  # 1 doctest failed
sage -t --long src/sage/plot/graphics.py  # 1 doctest failed
sage -t --long src/sage/plot/plot.py  # 1 doctest failed
sage -t --long src/sage/rings/polynomial/polynomial_real_mpfr_dense.pyx  # 1 doctest failed
sage -t --long src/sage/schemes/elliptic_curves/lseries_ell.py  # 1 doctest failed
sage -t --long src/sage/structure/coerce.pyx  # 1 doctest failed

one instance of numerical noise (lseries_ell.py) and the rest is unexpected warning messages from numpy. Needs to be investigated but everything is functional.

comment:182 follow-up: Changed 19 months ago by dimpase

on freebsd/clang numpy produces warnings like

        numpy.float32('1.5') * x
    :
    RuntimeWarning: invalid value encountered in multiply
    1.50000000000000*x

(this one from src/sage/plot/plot.py) Do you get the same?

The biggest difference is that on freebsd I also get problems with sympow, it does not fully work on some curves...

Last edited 19 months ago by dimpase (previous) (diff)

comment:183 Changed 19 months ago by fbissey

Yes, same. sage/structure/coerce.pyx same, sage/rings/polynomial/polynomial_real_mpfr_dense.pyx same. The other ones are

RuntimeWarning: invalid value encountered in greater_equal

comment:184 Changed 19 months ago by git

  • Commit changed from fca84601cfe2c9dc0288950c671c53c64bcde411 to d472cca9e04c100996009d74ec42173202e49168

Branch pushed to git repo; I updated commit sha1. New commits:

d472ccaMerge branch 'develop' into clang-start

comment:185 Changed 19 months ago by git

  • Commit changed from d472cca9e04c100996009d74ec42173202e49168 to eab0de59d8093c77f2e13cd2d96b4ea09f571429

Branch pushed to git repo; I updated commit sha1. New commits:

eab0de5generate new configure tarball

comment:186 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:187 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:188 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:189 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:190 Changed 19 months ago by jhpalmieri

It is exciting to see this progress, and it is exciting to have the gcc build take only 7 minutes on my Mac, since it only has to build fortran.

comment:191 follow-up: Changed 19 months ago by jhpalmieri

Do we have any good benchmarks? I don't know of any reason why Sage built this way should differ in performance compared to Sage built with gcc, but we should probably test it.

comment:192 in reply to: ↑ 191 Changed 19 months ago by fbissey

Replying to jhpalmieri:

Do we have any good benchmarks? I don't know of any reason why Sage built this way should differ in performance compared to Sage built with gcc, but we should probably test it.

I don't know that we have. But moving to clang on OS X just makes sense. We had to include numerous work around for gcc itself, and some package are already compiled with clang on a case by case basis. This is because we will continue having problems with gcc until either

  • gcc adopt the same extensions as clang
  • apple decides it isn't a good thing to use clang extension in their libc headers and fixes it

Neither is happening anytime soon as far as I am aware. It is increasingly difficult to build a big project on OS X with gcc/g++. The general mood is to just bite the bullet and move to clang.

Last edited 19 months ago by fbissey (previous) (diff)

comment:193 Changed 19 months ago by jhpalmieri

Don't misunderstand me, I am happy to see the change. We should make sure it doesn't lead to any regressions, though.

comment:194 Changed 19 months ago by fbissey

OK, benchmarking has always been a thorny issue. I don't believe we have one apart from doctesting time.

comment:195 follow-up: Changed 19 months ago by jhpalmieri

I tried some ad hoc testing, and I don't see any speed difference between the two builds. I don't know if anyone wants to try anything systematic.

comment:196 in reply to: ↑ 195 ; follow-up: Changed 19 months ago by dimpase

Replying to jhpalmieri:

I tried some ad hoc testing, and I don't see any speed difference between the two builds. I don't know if anyone wants to try anything systematic.

Internet has benchmarking stuff for clang vs gcc, in particular as far as HPC things are concerned. Give or take, they are largely on par. (One nice thing about clang is that it compiles and links faster than gcc, this is something seemingly well-known).

By the way, what is the assembler used by clang on OSX? Is it built-in, or external?

comment:197 in reply to: ↑ 196 Changed 19 months ago by jhpalmieri

Replying to dimpase:

By the way, what is the assembler used by clang on OSX? Is it built-in, or external?

I can check. How do I identify the assembler?

comment:198 follow-up: Changed 19 months ago by jdemeyer

I seem to recall that clang has a built-in assembler. But don't quote me on this...

comment:199 Changed 19 months ago by jhpalmieri

The man page says:

Clang also supports the use of an integrated assembler

and lists as options

-integrated-as, -no-integrated-as
       Used to enable and disable, respectively,  the  use  of  the  integrated
       assembler.  Whether  the integrated assembler is on by default is target
       dependent.

comment:200 in reply to: ↑ 198 Changed 19 months ago by fbissey

Interesting

Mirage:~ fbissey$ clang -cc1 --help | grep assembler
  -emit-llvm              Use the LLVM representation for assembler and object files
  -fno-integrated-as      Disable the integrated assembler
  -massembler-fatal-warnings
                          Make assembler warnings fatal

So an option to disabler internal assembler is lister but not one to enable it, which suggest that it is the default.

comment:201 Changed 19 months ago by jhpalmieri

I'm building Sage with SAGE_CHECK=yes on OS X, and I am seeing some failures of test suites. For some packages (ntl, gf2x) I see failures when building with gcc or with clang, but with zeromq, I see a new failure just with clang:

Invalid argument (stream_engine.cpp:94)
/bin/sh: line 1: 58001 Abort trap: 6           ${dir}$tst
FAIL: test_shutdown_stress

comment:202 Changed 19 months ago by fbissey

Yes, I haven't done that extensively with my little laptop. I will investigate zeromq - I think it is due a version bump anyway, our pyzmq complains that zeromq is too old. Version bump could take care of the failure, or not.

comment:203 follow-up: Changed 19 months ago by fbissey

OK, very easy solution to that.

[zeromq-4.0.5] Assertion failed: (rc == 0), function main, file test_abstract_ipc.cpp, line 31.
[zeromq-4.0.5] /bin/sh: line 1:  8898 Abort trap: 6           ${dir}$tst
[zeromq-4.0.5] XFAIL: test_abstract_ipc
[zeromq-4.0.5] PASS: test_many_sockets
[zeromq-4.0.5] PASS: test_shutdown_stress
[zeromq-4.0.5] PASS: test_pair_ipc
[zeromq-4.0.5] PASS: test_reqrep_ipc
[zeromq-4.0.5] PASS: test_timeo
[zeromq-4.0.5] PASS: test_fork
[zeromq-4.0.5] =====================================================
[zeromq-4.0.5] All 43 tests behaved as expected (1 expected failure)
[zeromq-4.0.5] =====================================================
[zeromq-4.0.5] Making check in tools
[zeromq-4.0.5] make[3]: Nothing to be done for `check'.
[zeromq-4.0.5] make[3]: Nothing to be done for `check-am'.
[zeromq-4.0.5] 
[zeromq-4.0.5] real	0m8.181s
[zeromq-4.0.5] user	0m0.985s
[zeromq-4.0.5] sys	0m1.320s
[zeromq-4.0.5] Deleting temporary build directory
[zeromq-4.0.5] /Users/fbissey/build/sage-clang/local/var/tmp/sage/build/zeromq-4.0.5
[zeromq-4.0.5] Finished installing zeromq-4.0.5.spkg

The gentoo ebuild for zeromq notes that upstream doesn't support testing in parallel. Just doing

diff --git a/build/pkgs/zeromq/spkg-check b/build/pkgs/zeromq/spkg-check
index 598280def1..3b36b91b55 100755
--- a/build/pkgs/zeromq/spkg-check
+++ b/build/pkgs/zeromq/spkg-check
@@ -6,5 +6,5 @@ if [ "$UNAME" = "Darwin" ]; then
 fi
 
 cd src
-$MAKE check
+$MAKE -j1 check
 

makes the test pass for me.

comment:204 follow-up: Changed 19 months ago by fbissey

I have looked into ntl and gf2x. I am assuming that it is in fact the same, or related, failure in both case. I think we should report it to upstream gf2x.

comment:205 follow-up: Changed 19 months ago by jhpalmieri

Is it the same as #18882?

comment:206 in reply to: ↑ 205 Changed 19 months ago by fbissey

Replying to jhpalmieri:

Is it the same as #18882?

Yes.

[gf2x-1.1.p1] -n 100x100 
[gf2x-1.1.p1] failed : '100 100 cca56aa6 76540123' != '100 100 2269c4fa 76540123'

comment:207 Changed 19 months ago by jhpalmieri

ccache also fails for me, both gcc and clang builds. I just opened #22836.

comment:208 follow-up: Changed 19 months ago by jhpalmieri

Another clang-specific failure on OS X: giac. On one hand, giac builds in 35sec with clang, as compared to 6m with gcc (!), but I get one test suite failure with clang:

51c51
< -sqrt(6)*I*sqrt(sqrt(13)+3),sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
---
> sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
FAIL: chk_fhan16

comment:209 Changed 19 months ago by jhpalmieri

Cython fails its test suite, with both clang and gcc, reported at #22166.

comment:210 Changed 19 months ago by jhpalmieri

Note that if a package fails its test suite with both gcc and clang, I am not at all saying that we need to deal with it before this ticket gets positively reviewed. I'm just reporting the current situation. When clang introduces new failures, we should investigate.

comment:211 in reply to: ↑ 203 Changed 19 months ago by jhpalmieri

Replying to fbissey:

OK, very easy solution to that.

[ snip ]

The gentoo ebuild for zeromq notes that upstream doesn't support testing in parallel. Just doing

diff --git a/build/pkgs/zeromq/spkg-check b/build/pkgs/zeromq/spkg-check
index 598280def1..3b36b91b55 100755
--- a/build/pkgs/zeromq/spkg-check
+++ b/build/pkgs/zeromq/spkg-check
@@ -6,5 +6,5 @@ if [ "$UNAME" = "Darwin" ]; then
 fi
 
 cd src
-$MAKE check
+$MAKE -j1 check
 

makes the test pass for me.

The test passed for me without any changes on a different machine, so it is intermittent. We should probably make this change anyway. See #22837. (I've made you the author and me the reviewer.)

comment:212 in reply to: ↑ 208 Changed 19 months ago by fbissey

Replying to jhpalmieri:

Another clang-specific failure on OS X: giac. On one hand, giac builds in 35sec with clang, as compared to 6m with gcc (!), but I get one test suite failure with clang:

51c51
< -sqrt(6)*I*sqrt(sqrt(13)+3),sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
---
> sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*I*sqrt(sqrt(13)+3),-sqrt(6)*sqrt(sqrt(13)-3),sqrt(6)*sqrt(sqrt(13)-3),0,
FAIL: chk_fhan16

Most interesting because it originates with https://github.com/sagemath/sage/blob/master/build/pkgs/giac/patches/cSolveorder-check.patch I am sure. And that patch is needed with gcc for pari-2.8+ (i.e. with pari 2.7 it is not needed).

comment:213 Changed 19 months ago by jhpalmieri

cysignals also fails its test suite. This is not fixed by #22695: that is, both the current version and the version in #22695 pass tests with gcc and fail with clang. The failures:

**********************************************************************
File "src/cysignals/tests.pyx", line 168, in tests.pyx
Failed example:
    test_sig_on()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 182, in tests.pyx
Failed example:
    test_sig_str()
Expected:
    Traceback (most recent call last):
    ...
    RuntimeError: Everything ok!
Got:
    Traceback (most recent call last):
      File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest tests.pyx[12]>", line 1, in <module>
        test_sig_str()
      File "src/cysignals/tests.pyx", line 189, in cysignals.tests.test_sig_str (build/src/cysignals/tests.c:2665)
        sig_str("Everything ok!")
      File "src/cysignals/signals.pyx", line 109, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1699)
        raise SignalError(s or "Segmentation fault")
    SignalError: Everything ok!
**********************************************************************
File "src/cysignals/tests.pyx", line 203, in tests.pyx
Failed example:
    test_sig_on_cython()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 220, in tests.pyx
Failed example:
    test_sig_on_cython_except()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 238, in tests.pyx
Failed example:
    test_sig_on_cython_except_all()
Expected:
    KeyboardInterrupt()
Got:
    SystemError('unknown signal number 0',)
**********************************************************************
File "src/cysignals/tests.pyx", line 267, in tests.pyx
Failed example:
    test_sig_check_inside_sig_on()
Expected:
    KeyboardInterrupt()
Got:
    RuntimeError('Aborted',)
Traceback (most recent call last):
  File "rundoctests.py", line 20, in <module>
    failures, _ = doctest.testfile(f, module_relative=False, optionflags=flags)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 2036, in testfile
    runner.run(test)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 1454, in run
    return self.__run(test, compileflags, out)
  File "/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta2/local/lib/python2.7/doctest.py", line 1315, in __run
    compileflags, 1) in test.globs
  File "src/cysignals/signals.pyx", line 252, in cysignals.signals.python_check_interrupt (build/src/cysignals/signals.c:2858)
    sig_check()
  File "src/cysignals/signals.pyx", line 97, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1307)
    raise KeyboardInterrupt
KeyboardInterrupt
make[3]: *** [check-doctest] Interrupt: 2

comment:214 Changed 19 months ago by jhpalmieri

eclib fails its test suite with clang but not with gcc. Looks like just numerical noise, I hope:

Testing smattest...
Testing comptest...
5c5
< AGM((3.125,4.25),(1,0)) = (2.07831617217326684,1.5633615899065958201)
---
> AGM((3.125,4.25),(1,0)) = (2.07831617217326684,1.56336158990659582)
14c14
< whose cube is    (2.9999999999999999998,3.9999999999999999999)
---
> whose cube is    (2.9999999999999999999,4)
16c16
< whose cube is    (2.9999999999999999997,3.9999999999999999999)
---
> whose cube is    (2.9999999999999999996,3.9999999999999999998)
make[4]: *** [check_procs] Error 1
make[3]: *** [check] Error 2
make[2]: *** [check-recursive] Error 1
Error: eclib's test suite failed to pass.

comment:215 Changed 19 months ago by fbissey

Hum, cysignals stuff looks serious, looks like some mapping is wrong. For eclib it really looks like just numerical noise.

comment:216 Changed 19 months ago by fbissey

I can reproduce the eclib failures on linux+clang.

comment:217 follow-ups: Changed 19 months ago by jhpalmieri

That's my complete list of failed test suites. Summarizing: clang-only failures for:

  • eclib: numerical noise, so hopefully minor
  • giac
  • cysignals

Failures with both clang and gcc:

  • ccache: #22836
  • gf2x and ntl (related?)
  • zeromq: intermittent, perhaps addressed at #22837 although I haven't tested thoroughly to see if that completely fixes the problem
  • cython: #22166

Oh, and I forgot to mention: python3 – does any python package ever pass its test suite with Sage?

comment:218 in reply to: ↑ 217 Changed 19 months ago by fbissey

Replying to jhpalmieri:

Oh, and I forgot to mention: python3 – does any python package ever pass its test suite with Sage?

I don't think we have in some time but Volker would have some ideas on that.

comment:219 Changed 19 months ago by fbissey

  • Description modified (diff)

I just finished a build of 8.0.beta2 with clang+linux, hacking my way through the pyzmq problem, and there is a truckload of doctest failures. However it is exclusively, as far as I can see through the mess, all done to a failure in sage.matrix.matrix_integer_dense.so. A nice isolated example is

sage -t --long src/sage/rings/qqbar.py
**********************************************************************
File "src/sage/rings/qqbar.py", line 237, in sage.rings.qqbar
Failed example:
    r.imag().minpoly()  # long time (10s on sage.math, 2013)
Exception raised:
    Traceback (most recent call last):
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 503, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 866, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.rings.qqbar[72]>", line 1, in <module>
        r.imag().minpoly()  # long time (10s on sage.math, 2013)
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/rings/qqbar.py", line 3403, in minpoly
        self._minimal_polynomial = self._descr.minpoly()
      File "/home/fbissey/sandbox/git-fork/sage-clang/local/lib/python2.7/site-packages/sage/rings/qqbar.py", line 6426, in minpoly
        self._minpoly = self._value.minpoly()
      File "sage/rings/number_field/number_field_element.pyx", line 4291, in sage.rings.number_field.number_field_element.NumberFieldElement_absolute.minpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/rings/number_field/number_field_element.cpp:40412)
        return self.charpoly(var, algorithm).radical() # square free part of charpoly
      File "sage/rings/number_field/number_field_element.pyx", line 4258, in sage.rings.number_field.number_field_element.NumberFieldElement_absolute.charpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/rings/number_field/number_field_element.cpp:40187)
        return R(self.matrix().charpoly())
      File "sage/matrix/matrix_rational_dense.pyx", line 945, in sage.matrix.matrix_rational_dense.Matrix_rational_dense.charpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_rational_dense.c:11214)
        f = A.charpoly(var, algorithm='linbox')
      File "sage/matrix/matrix_integer_dense.pyx", line 1251, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.charpoly (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_integer_dense.c:13209)
        g = self._charpoly_linbox(var)
      File "sage/matrix/matrix_integer_dense.pyx", line 1304, in sage.matrix.matrix_integer_dense.Matrix_integer_dense._charpoly_linbox (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_integer_dense.c:14045)
        return self._poly_linbox(var=var, typ='charpoly')
      File "sage/matrix/matrix_integer_dense.pyx", line 1327, in sage.matrix.matrix_integer_dense.Matrix_integer_dense._poly_linbox (/home/fbissey/sandbox/git-fork/sage-clang/src/build/cythonized/sage/matrix/matrix_integer_dense.c:14435)
        sig_on()
      File "src/cysignals/signals.pyx", line 109, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1699)
    SignalError: Segmentation fault
**********************************************************************
1 item had failures:

comment:220 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:221 in reply to: ↑ 217 ; follow-up: Changed 19 months ago by jdemeyer

Replying to jhpalmieri:

That's my complete list of failed test suites. Summarizing: clang-only failures for:

  • cysignals

I just had some "fun" debugging this. Turns out it is a 9-year old(!) LLVM bug: https://bugs.llvm.org/show_bug.cgi?id=965

There is a work-around in #22691.

comment:222 in reply to: ↑ 221 Changed 19 months ago by jhpalmieri

Replying to jdemeyer:

Replying to jhpalmieri:

That's my complete list of failed test suites. Summarizing: clang-only failures for:

  • cysignals

I just had some "fun" debugging this. Turns out it is a 9-year old(!) LLVM bug: https://bugs.llvm.org/show_bug.cgi?id=965

There is a work-around in #22691.

Great! That works for me on OS X.

comment:223 Changed 19 months ago by git

  • Commit changed from eab0de59d8093c77f2e13cd2d96b4ea09f571429 to caa9ac9bf760aba1451e25f8751569fe042f3d94

Branch pushed to git repo; I updated commit sha1. New commits:

ec06d97Merge branch 'develop' into clang-start
caa9ac9bumping configure tarball

comment:224 Changed 19 months ago by fbissey

  • Description modified (diff)
  • Milestone changed from sage-7.6 to sage-8.0

comment:225 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:226 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:227 in reply to: ↑ 182 Changed 19 months ago by rws

Replying to dimpase:

The biggest difference is that on freebsd I also get problems with sympow, it does not fully work on some curves...

This may be the same as what I get on OpenSuSE, invalid pointers in sympow leading to runtime errors and failing to compute the analytic rank. I opened #22862.

comment:228 Changed 19 months ago by jhpalmieri

Following up on a comment from #22756: right now we build python2, python3 (as of #22756), curl, psutil, and R using clang rather than gcc, because of troubles with gcc.

With this branch, if a user on OS X happens to have installed gcc (for example, version 5.4.0 in /usr/local/bin), then those packages will break if you use that version of gcc to compile them. So if here or on a followup ticket, we are hoping to simplify the logic in some of the spkg-install scripts, we have to be careful. We either need to keep the parts of the script that say

if [ "$UNAME" = "Darwin" ] && [ $MACOSX_VERSION -ge 16 ]; then
    echo "OS X 10.$[$MACOSX_VERSION-4] Building with clang."
    CC=clang
fi

or we need to force the user to compile everything with clang even if they have a good version of gcc installed.

comment:229 Changed 19 months ago by fbissey

Well, I think I will want to force clang. At the moment we keep adding package exceptions to use clang - and I mean clang but not clang++. This is important, the libc is provided by the system for both gcc and clang, so apart from a few isolated exceptions you can mix c compilers.

Now if we happen to want to make a clang++ exception for a package, and it is not stand alone - ie. it provides a library that will be consumed by sage or other packages, then we will have no choice but convert all the dependency chain to clang++. This is because clang++ and g++ each have their own runtimes and you cannot mix them up (#22798). Technically they should be ABI compatible (provided the proper incantation), the reality is that it is madness.

comment:230 Changed 19 months ago by git

  • Commit changed from caa9ac9bf760aba1451e25f8751569fe042f3d94 to 678988c254e96125cd6945cf58a91be2519f3f32

Branch pushed to git repo; I updated commit sha1. New commits:

2dec934Merge branch 'develop' into clang-start
678988cbump configure tarball

comment:231 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:232 Changed 19 months ago by dimpase

isn't it important to explicitly use CC=clang even on OSX? Note that gcc and g++ in Xcode are meant to try to be gcc-compatible:

$ g++ -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Volumes/Sage/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

note --with-gxx-include-dir, absent in cc configuration.

comment:233 Changed 19 months ago by fbissey

I don't think so. Yes the gcc/g++ compiler drivers gives a bit more info than cc or clang

Mirage:~ fbissey$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Mirage:~ fbissey$ cc -v
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
Mirage:~ fbissey$ clang -v
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

but I don't think it is that important. Whether you set CC=lang or not, autotools still thinks clang is gcc. We have to use an extra non default macro to figure out it is not gcc. I don't think there is a difference in output whether I set it or not.

comment:234 Changed 19 months ago by dimpase

I did set CC and CXX, and on OSX get

[gcc-5.4.0] checking for x86_64-apple-darwin16.5.0-gcc... /Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/xgcc -B/Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/ -B/Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/bin/ -B/Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/lib/ -isystem /Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/include -isystem /Volumes/Sage/sage/local/x86_64-apple-darwin16.5.0/sys-include   
[gcc-5.4.0] checking for suffix of object files... configure: error: in `/Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/x86_64-apple-darwin16.5.0/libgcc':
[gcc-5.4.0] configure: error: cannot compute suffix of object files: cannot compile
[gcc-5.4.0] See `config.log' for more details.
[gcc-5.4.0] make[4]: *** [configure-target-libgcc] Error 1

Changed 19 months ago by dimpase

OSX gcc spkg failure

comment:235 Changed 19 months ago by fbissey

Interesting. I don't need the log, I have seen that error so many times in many different context. Basically the gcc you built in the previous stage is not "working", usually because it cannot find gmp/mpfr/mpc or find the right one because of {DY,}LD_LIBRARY_PATH settings or sometimes lack thereof. It should be manifest in the output of

otool -L /Volumes/Sage/sage/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/xgcc

comment:236 Changed 19 months ago by fbissey

Very interesting... Setting CC=clang CXX=clang++ works no problems. Setting CC=cc CXX=c++ leads to the above error and problem in finding libz is the culprit this time.

/Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/cc1:
	/usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
	/Users/fbissey/build/sage-build/local/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
	/Users/fbissey/build/sage-build/local/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.5.0)
	/Users/fbissey/build/sage-build/local/lib/libgmp.16.dylib (compatibility version 23.0.0, current version 23.2.0)
	libz.so.1.2.8 (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.51.1)

and from the right config.log

configure:3464: /Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/xgcc -B/Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/ -B/Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/bin/ -B/Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/lib/ -isystem /Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/include -isystem /Users/fbissey/build/sage-build/local/x86_64-apple-darwin16.5.0/sys-include    -o conftest -g -O2   conftest.c  >&5
dyld: Library not loaded: libz.so.1.2.8
  Referenced from: /Users/fbissey/build/sage-build/local/var/tmp/sage/build/gcc-5.4.0/gcc-build/./gcc/cc1
  Reason: image not found
xgcc: internal compiler error: Abort trap: 6 (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.

Of course the very curious bit here is that libz.so has been built instead of libz.dylib and it doesn't have the proper install_name set which leads to the problem. So the real problem is that zlib is built funny when the compiler is set to cc. I expect it doesn't use autotools....

comment:237 Changed 19 months ago by fbissey

zlib has a hacked configure script and we get the following results

Mirage:zlib-1.2.8 fbissey$ CC=cc ./configure
Checking for shared library support...
Building shared library libz.so.1.2.8 with cc.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Mirage:zlib-1.2.8 fbissey$ CC=clang ./configure
Checking for shared library support...
Building shared library libz.1.2.8.dylib with clang.
Checking for off64_t... No.
Checking for fseeko... Yes.
Checking for strerror... Yes.
Checking for unistd.h... Yes.
Checking for stdarg.h... Yes.
Checking whether to use vs[n]printf() or s[n]printf()... using vs[n]printf().
Checking for vsnprintf() in stdio.h... Yes.
Checking for return value of vsnprintf()... Yes.
Checking for attribute(visibility) support... Yes.

So there is detection gap between compiler name and platform somewhere.

comment:238 Changed 19 months ago by fbissey

Yes the configure script for zlib is broken in that particular case

case "$cc" in
  *gcc*) gcc=1 ;;
  *clang*) gcc=1 ;;
esac
case `$cc -v 2>&1` in
  *gcc*) gcc=1 ;;
esac

Earlier we have gcc=0, so in theory when CC is either gcc or clang the variable gcc is set to 1, and you are sent on a branch where darwin is identified and the right linking flags are set. However if you set CC=cc the first case statement doesn't do anything, and then the second doesn't either since

Mirage:zlib-1.2.8 fbissey$ cc -v
Apple LLVM version 8.1.0 (clang-802.0.42)
Target: x86_64-apple-darwin16.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

doesn't mention gcc anywhere.

comment:239 follow-up: Changed 19 months ago by fbissey

Upstream fixed the configuration problem (same way I did for experimental purpose), upgrading to 1.2.11 is recommended in another ticket if you think it is essential.

comment:240 in reply to: ↑ 239 Changed 19 months ago by dimpase

Replying to fbissey:

Upstream fixed the configuration problem (same way I did for experimental purpose), upgrading to 1.2.11 is recommended in another ticket if you think it is essential.

the update is on #22917 - let's get it done to avoid possible confusion.

comment:241 Changed 19 months ago by git

  • Commit changed from 678988c254e96125cd6945cf58a91be2519f3f32 to 12cf61eb6a6a42478cac33bee4cc03030213b0a1

Branch pushed to git repo; I updated commit sha1. New commits:

12cf61eMerge branch 'develop' into clang-start

comment:242 Changed 19 months ago by fbissey

  • Description modified (diff)

comment:243 Changed 18 months ago by git

  • Commit changed from 12cf61eb6a6a42478cac33bee4cc03030213b0a1 to 38f93525fd35e29c143c7f1fe244d2cb293a1208

Branch pushed to git repo; I updated commit sha1. New commits:

cec66dfMerge branch 'develop' into clang-start
66973afPaul's workaround from #22799
b3d23f9Merge branch 'develop' into clangtrouble
811ba22Make the work around only compile with clang
67fa2fabump version
c881b5dMerge branch 'clangtrouble' into clang-start
38f9352increment configure tarball

comment:244 Changed 18 months ago by fbissey

  • Dependencies set to #22799

comment:245 Changed 18 months ago by fbissey

  • Description modified (diff)

comment:246 Changed 18 months ago by fbissey

A little word about future plans to finish this now that #22799 seems to have a solution.

  • It is now possible to build, and doctest sage itself on OS X with clang building only gfortran with this ticket.
  • Similarly on FreeBSD.
  • To build with clang on linux we need to patch python (both 2.7 and 3.x) for http://bugs.python.org/issue25229 for which I should make a ticket.
  • Further on linux which usually default to gcc we want something like #22646, otherwise you have to run sage like CC=clang CXX=clang++ ./sage if you want to avoid problems like the one described at #22678.
  • Some packages do not pass their test suite with clang.

After much thinking, while I think the default on OS X should be to use clang, I don't think we should preclude building with gcc, especially if people bring their own gcc. So far the only problems we have seen are in C code. Not C++ and after reflexion I don't think it is going to change. So building with gcc/g++ with clang exception for key packages should work. But apart from those, I think the support for gcc on OS X should become minimal over time.

You could even imagine doing some compile with gcc/clang++/gfortran or clang/g++/gfortran so long as we fix python use of the C compiler to compile C++ code. https://bugs.python.org/issue1222585

So the plan forward:

  • Fix package testsuite as much as possible - sometimes it will be hard. eclib's tests need to learn about tolerance for example.
  • Base future work on #22646
  • Make the building of gfortran only, more generic. That is, not relying on a variable telling you the compiler is clang. For that matter we can try to extend this to the case where the platform already has a gcc/g++ compiler but only lacks a fortran compiler. But in a general manner not exposing to the rest of the system that you are not using gcc by a variable like IS_CLANG.

comment:247 Changed 18 months ago by fbissey

  • Description modified (diff)

comment:248 in reply to: ↑ 204 Changed 18 months ago by fbissey

Replying to fbissey:

I have looked into ntl and gf2x. I am assuming that it is in fact the same, or related, failure in both case. I think we should report it to upstream gf2x.

Confirmed that ntl testsuite failure is due to gf2x. Once I solve the failure of gf2x by not running make tune-toom (https://trac.sagemath.org/ticket/18882#comment:5) the ntl testsuite complete successfully.

comment:249 follow-up: Changed 18 months ago by dimpase

would you go with having compilers are parameters to configure, or rather stay with env. variables? (IMHO autotools do not offer good support for choosing compilers...)

comment:250 Changed 18 months ago by fbissey

Well autotools finds a compiler that works. You can over-ride that by parameters. Note that #22646 makes that choice of compiler stick to runtime unless sage builds its own.

Autotools is system based on testing and finding capabilities of what you have rather than sticking to a brand. Which is what you should wish for. That being said if you don't set CC, AC_PROG_C will go through a default list (which you can override). On OS X, with the current branch it finds Apple's gcc which is really clang and everything is fine.

As far as I am concerned doing CC=.. ./configure.. is not very different to doing something like ./configure --with-c-compiler=....

But we could be enforcing things more at the AC_PROG_{C,CXX,FC} point and even do it by OS. On linux only look for gcc and clang in that order, on OS X and freebsd look for clang and gcc in that order. That's bit harsh but can still be discarded in the event you set CC (I think).

comment:251 in reply to: ↑ 249 Changed 18 months ago by mkoeppe

Replying to dimpase:

would you go with having compilers are parameters to configure, or rather stay with env. variables? (IMHO autotools do not offer good support for choosing compilers...)

I'd strongly suggest to stick with using environment variables (note that it is best to pass them to configure as arguments (./configure CC=...), simply because that's the standard way to select a compiler with autotools packages. Best not to invent new sage-specific ways.

comment:252 follow-up: Changed 18 months ago by jhpalmieri

Whatever happens, we need to be careful not to build certain packages on OS X using gcc: in my experience, with Sage's gcc or my own, some packages (like python and R) fail, but they succeed using clang. So whether we use autotools or configure options or something else, we might have to force clang for some packages if we don't force it in general.

I guess we could let the user shoot themselves in the foot by explicitly setting CC etc., but the default settings should work whether or not there is a recent version of gcc in /usr/local/bin, for example.

comment:253 in reply to: ↑ 252 Changed 18 months ago by fbissey

Replying to jhpalmieri:

Whatever happens, we need to be careful not to build certain packages on OS X using gcc: in my experience, with Sage's gcc or my own, some packages (like python and R) fail, but they succeed using clang. So whether we use autotools or configure options or something else, we might have to force clang for some packages if we don't force it in general.

Yes, I think we'll still make sure the packages that needs clang on OS X gets it enforced at the spkg-install level.

I guess we could let the user shoot themselves in the foot by explicitly setting CC etc., but the default settings should work whether or not there is a recent version of gcc in /usr/local/bin, for example.

Yes. By default it should take a well supported configuration for the platform. But users should be free to shoot themselves in the foot - best way to learn things, really.

Last edited 18 months ago by fbissey (previous) (diff)

comment:254 Changed 18 months ago by git

  • Commit changed from 38f93525fd35e29c143c7f1fe244d2cb293a1208 to 40d907acaf4ad226dad45464e648989674176b82

Branch pushed to git repo; I updated commit sha1. New commits:

e4384afMerge branch 'develop' into clang-start
40d907abump configure tarball

comment:255 Changed 18 months ago by fbissey

  • Description modified (diff)

comment:256 Changed 17 months ago by jhpalmieri

Is there anything I can do to help move this ticket along? What needs to be done now?

comment:257 Changed 17 months ago by fbissey

Sorry, I am a bit bogged down in stuff and that has shifted to a lower priority than say, python3. So getting #23046 reviewed would be interesting because you can then test clang on a linux machine. Another ticket in need of attention is #22646 which will need some rebasing for the configure tarball. #22646 is important because it help your compiler to "stick" if you use a compiler than is not in the PATH or not the expected default compiler. Because it touches some of the same files as this ticket, I would very much want it included first.

The next step is to rework the logic so we can trigger the build of gfortran without the building of gcc - as in the attached branch - but without using IS_CLANG. I would like it to be more generic if just gfortran is missing and that includes the case of clang on OS X by default. Bonus point if we can get out of rebuilding mpir, mpfr and mpc after gcc when you build just gfortran.

comment:258 Changed 17 months ago by git

  • Commit changed from 40d907acaf4ad226dad45464e648989674176b82 to 1b2d84cbf74bcb242e13bd7ce24d14a5b65eeb35

Branch pushed to git repo; I updated commit sha1. New commits:

1b2d84cMerge branch 'develop' into clang-start

comment:259 Changed 17 months ago by fbissey

  • Description modified (diff)

comment:260 Changed 17 months ago by git

  • Commit changed from 1b2d84cbf74bcb242e13bd7ce24d14a5b65eeb35 to fb0ff15ef6cdacbac30326d1f40d75b017d78200

Branch pushed to git repo; I updated commit sha1. New commits:

fb0ff15new configure tarball

comment:261 Changed 17 months ago by fbissey

I think the solution to this will have to be a gfortran package as part of all-toolchain that can only be installed when gcc isn't.

comment:262 Changed 17 months ago by fbissey

Say, what is the oldest OS X we support?

comment:263 Changed 17 months ago by fbissey

  • Branch changed from u/fbissey/clang-start to u/fbissey/clang-progress
  • Commit changed from fb0ff15ef6cdacbac30326d1f40d75b017d78200 to 8d8b174bc99d61e43937d7032ba604f2c75738aa
  • Dependencies changed from #22799 to #22799 #22646

New commits:

483106aFirst draft of moving compiler settings to sage-env-config
d6e87c8Merge branch 'develop' into compiler
6048dffForgot to commit the new configure tarball
f7eff64Merge branch 'develop' into compiler
945e824bump config tarball
32a0b00one AC_SUBST per variable only :(
11dc29aconfigure/automake doesn't like variables only defined in conditionals
ef9f1b3missing "fi" in sage-env-config.in
8d8b174Add a new gfortran spkg and use it when gfortran is not found and you don't have to build the rest of gcc

comment:264 Changed 17 months ago by fbissey

OK, revamped branch. This new branch focuses on being able to build gfortran when not found and allow CC/CXX to be clang. It now fully depends on #22646.

comment:265 Changed 17 months ago by fbissey

And the current branch is not yet working on OS X. Figuring out why.....

comment:266 Changed 17 months ago by git

  • Commit changed from 8d8b174bc99d61e43937d7032ba604f2c75738aa to 40faa0f4c558522cfcd8e1288d8b1d89484741d5

Branch pushed to git repo; I updated commit sha1. New commits:

40faa0fupdating GFC check section for gfortran only potential install

comment:267 Changed 17 months ago by git

  • Commit changed from 40faa0f4c558522cfcd8e1288d8b1d89484741d5 to bd8c1afda6d9b02a90cf522a0530e1ce3c62adb8

Branch pushed to git repo; I updated commit sha1. New commits:

bd8c1afnew configure checksums

comment:268 Changed 17 months ago by git

  • Commit changed from bd8c1afda6d9b02a90cf522a0530e1ce3c62adb8 to 328718020e400963ee15392740c59735e4165139

Branch pushed to git repo; I updated commit sha1. New commits:

3287180Do not install a C compiler or cpp with the gfortran spkg

comment:269 Changed 17 months ago by git

  • Commit changed from 328718020e400963ee15392740c59735e4165139 to 77107d850166c7b872c872512aaf55e69bb53bbf

Branch pushed to git repo; I updated commit sha1. New commits:

23c32a4Small clean up and refining
77107d8update config tarball checksum

comment:270 Changed 17 months ago by fbissey

OK, people interested should give this branch some trashing - both at the code comment level and testing level.

comment:271 follow-up: Changed 17 months ago by jhpalmieri

When the gfortran package is built on OS X, I also end up with these in local/bin: gcc-ar, gcc-nm, gcc-ranlib, x86_64-apple-darwin16.6.0-gcc, etc. Is that a problem?

comment:272 in reply to: ↑ 271 Changed 17 months ago by fbissey

Replying to jhpalmieri:

When the gfortran package is built on OS X, I also end up with these in local/bin: gcc-ar, gcc-nm, gcc-ranlib, x86_64-apple-darwin16.6.0-gcc, etc. Is that a problem?

It shouldn't be. I left the stuff that had a prefix. I only removed the non-bootstraped gcc and cpp binaries that are installed. I believe the rest should be ok to be left alone. I should inspect the gfortran binary package produced upstream. We could be a bit cleaner for all I know.

comment:273 Changed 17 months ago by fbissey

I should say a bit more I guess. You ended up with that stuff too with the old branch. While I only want gfortran the build system cannot build it without compiling gcc first - at least I haven't found a way to eliminate that step. I am not even sure that's currently desirable. So a gcc is produced to build gfortran and it is installed.

If you know a way to only install gfortran (even if it still builds a gcc), I am all ears. I will have a look in case it is easy to spot.

comment:274 Changed 17 months ago by git

  • Commit changed from 77107d850166c7b872c872512aaf55e69bb53bbf to a6db502d6762eda8cbbfd8e13f6582d8c8ad9501

Branch pushed to git repo; I updated commit sha1. New commits:

a6db502make some files in the gfortran pkg symlinks to the files in the gcc pkg so that that versions and dependencies are changed at the same time.

comment:275 Changed 17 months ago by fbissey

Couldn't find a way to install gfortran separately in gcc's makefiles. In the meantime I made some files in the gfortran folder link to the gcc folder. gcc version bumps are now in sync with changes to gfortran. One would still have to check spkg-install but it makes things easier.

In some ways the work here got a little bit of scope creep. Kind of of a mix of unintended consequences and some thoughts on trying to be generic.:

  • As tested by Vincent Delecroix, this ticket allows the version of gfortran to be decoupled from the version of gcc used. It is of course non sense to check the version of the C/C++ compiler matches the fortran one when they may not be from the same vendor. However it could have been kept for gcc specifically. Earlier versions of the branch did.
  • I switched from positively checking for clang to checking "is this really gcc?". clang of course pretends to be gcc but so does icc I think. The branch in its current state would allow one to use a compiler that pretends to be gcc to AC_PROG_CC but isn't. This really needs #23046 to work on linux, probably not just for clang.

The ticket keeps the requirement for gfortran but I think that could be relaxed in the future. The real hard requirement is for R. Some R packages don't properly autotool their fortran which means they rely on fortran mangling to be "function_". Any fortran compiler that default to that or can easily be made to do that will do.

comment:276 Changed 17 months ago by fbissey

  • Status changed from new to needs_review

comment:277 Changed 17 months ago by jhpalmieri

Here is a report about building on OS X:

  • if I have my own gfortran present (but no gcc), it builds and passes tests. It does not build gfortran.
  • if I have neither gfortran nor gcc, it builds and passes tests. It builds gfortran, of course.
  • if I have my own gcc (version 6.3.0), I got an error building libfplll:
    	/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -DFPLLL_DEFAULT_STRATEGY_PATH=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies/default.json\" -I./.. -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -I/include  -fPIC -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/lib -O3 -MT pruner.lo -MD -MP -MF $depbase.Tpo -c -o pruner.lo pruner.cpp &&\
    	mv -f $depbase.Tpo $depbase.Plo
    libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -DFPLLL_DEFAULT_STRATEGY_PATH=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies\" -DFPLLL_DEFAULT_STRATEGY=\"/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/share/fplll/strategies/default.json\" -I./.. -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -I/include -fPIC -I/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/include/ -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-8.0.beta11/local/lib -O3 -MT pruner.lo -MD -MP -MF .deps/pruner.Tpo -c pruner.cpp  -fno-common -DPIC -o .libs/pruner.o
    pruner.cpp: In function 'FT fplll::Pruner<FT>::single_enum_cost(const evec&, std::vector<double>*) [with FT = fplll::FP_NR<long double>]':
    pruner.cpp:247:11: internal compiler error: Segmentation fault: 11
     inline FT Pruner<FT>::single_enum_cost(/*i*/ const evec &b, vector<double> *detailed_cost)
               ^~~~~~~~~~
    
    pruner.cpp:247:11: internal compiler error: Abort trap: 6
    g++: internal compiler error: Abort trap: 6 (program cc1plus)
    Please submit a full bug report,
    with preprocessed source if appropriate.
    See <https://github.com/Homebrew/homebrew-core/issues> for instructions.
    make[6]: *** [pruner.lo] Error 1
    make[6]: *** Waiting for unfinished jobs....
    
    Maybe my version of gcc is broken. Any other ideas of what could be wrong?

comment:278 Changed 17 months ago by fbissey

Looks suspiciously like some recent changes to fplll. Let me check...

comment:279 Changed 17 months ago by fbissey

OK not what I was thinking. ICE are bad, the question is whether it is something from homebrew or upstream gcc. Does homebrew happen to have fplll in one of their repos? It would be interesting to know if a homebrew version exhibits the same problem.

Alternatively I/you could also try with gcc/gfortran from http://hpc.sourceforge.net/

comment:280 Changed 17 months ago by git

  • Commit changed from a6db502d6762eda8cbbfd8e13f6582d8c8ad9501 to 777a393a2344773c293595d00b469b5bb4115b70

Branch pushed to git repo; I updated commit sha1. New commits:

777a393updating configure tarball

comment:281 Changed 17 months ago by fbissey

I am trying gcc-7.1 from hpc.sourceforge.net. libfplll compiled with it. I am using gcc/g++/gfortran from that tarball. Checking with someone I know I am surprised you are on gcc-6.3.0 with homebrew. They have 7.1.0 now. Are you working on the polymake package (which apparently doesn't like gcc-7.1)?

comment:282 follow-up: Changed 17 months ago by jhpalmieri

I built and installed gcc 5.4.0 and got errors building libfplll and givaro:

ld: symbol(s) not found for architecture x86_64

Maybe I'm building gcc wrong?

I'm not using polymake, by the way. I haven't tried the version from hpc.sourceforge.net because if it breaks things, I don't know how hard it will be to uninstall.

comment:283 in reply to: ↑ 282 Changed 17 months ago by fbissey

Replying to jhpalmieri:

I built and installed gcc 5.4.0 and got errors building libfplll and givaro:

ld: symbol(s) not found for architecture x86_64

Maybe I'm building gcc wrong?

That's a classic message. Usually something slightly wrong in the setup. A bigger chunk of the failure may jog my memory.

I'm not using polymake, by the way. I haven't tried the version from hpc.sourceforge.net because if it breaks things, I don't know how hard it will be to uninstall.

It will be a pain to uninstall. Well not too bad so long as you keep the original tarball to have the list of files installed. It also helps that I don't have much in /usr/local in the first place. But I hate things like that, that insists they need to be installed in /usr/local. Whenever I try homebrew, I specifically do not follow their recommendation about linking in /usr/local - an invitation to disaster if you have more than one prefixed environment around that you want to be able to switch to and from.

comment:284 Changed 17 months ago by jhpalmieri

Here's more of the error message from givaro:

Undefined symbols for architecture x86_64:
  "std::ctype<char>::_M_widen_init() const", referenced from:
      Givaro::Rationel::Rationel(float, Givaro::Rationel::reduceFlag) in libgmppp.a(gmp++_rat_cstor.o)
      Givaro::Rationel::Rationel(double, Givaro::Rationel::reduceFlag) in libgmppp.a(gmp++_rat_cstor.o)
      Givaro::GivMMInfo::print(std::basic_ostream<char, std::char_traits<char> >&) const in libgivmemory.a(givaromm.o)
      Givaro::Rational::ratrecon(Givaro::Integer const&, Givaro::Integer const&, Givaro::Integer const&, bool) in libgivrational.a(givratreconstruct.o)
      Givaro::GivaroMain::DisplayVersion(std::basic_ostream<char, std::char_traits<char> >&) in libgivsystem.a(givinit.o)
      Givaro::Timer::print(std::basic_ostream<char, std::char_traits<char> >&) const in libgivsystem.a(givtimer.o)
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::swap(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
      Givaro::Indeter::operator=(Givaro::Indeter const&) in libgivpoly1.a(givindeter.o)
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)", referenced from:
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.21] in libgmppp.a(gmp++_rat_io.o)
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.51] in libgmppp.a(gmp++_int_misc.o)
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) [clone .isra.19] in libgivrational.a(givratcstor.o)
  "std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::_M_sync(char*, unsigned long, unsigned long)", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  "std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_istringstream()", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  "std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  "std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)", referenced from:
      Givaro::logp(Givaro::Integer const&, Givaro::Integer const&) in libgmppp.a(gmp++_int_misc.o)
  "std::__detail::_List_node_base::_M_unhook()", referenced from:
      Givaro::logp(Givaro::Integer const&, Givaro::Integer const&) in libgmppp.a(gmp++_int_misc.o)
  "std::basic_istream<char, std::char_traits<char> >& std::operator>><char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)", referenced from:
      Givaro::operator>>(std::basic_istream<char, std::char_traits<char> >&, Givaro::Indeter&) in libgivpoly1.a(givindeter.o)
  "VTT for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  "VTT for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  "vtable for std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for std::__cxx11::basic_istringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rational::Rational(char const*) in libgivrational.a(givratcstor.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >", referenced from:
      Givaro::Rationel::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_rat_io.o)
      Givaro::Integer::operator std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >[abi:cxx11]() const in libgmppp.a(gmp++_int_misc.o)
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status

comment:285 Changed 17 months ago by jhpalmieri

And from libpflll:

Undefined symbols for architecture x86_64:
  "std::ctype<char>::_M_widen_init() const", referenced from:
      fplll::shortest_vector_ex(fplll::ZZ_mat<__mpz_struct [1]>&, std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >&, fplll::SVPMethod, std::vector<double, std::allocator<double> > const&, int, fplll::EvaluatorMode, long long&, std::vector<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >, std::allocator<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > > > >*, std::vector<double, std::allocator<double> >*, std::vector<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >, std::allocator<std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > > > >*, std::vector<double, std::allocator<double> >*, int) [clone .constprop.282] in fplll.o
      fplll::closest_vector(fplll::ZZ_mat<__mpz_struct [1]>&, std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > > const&, std::vector<fplll::Z_NR<__mpz_struct [1]>, std::allocator<fplll::Z_NR<__mpz_struct [1]> > >&, int, int) in fplll.o
      fplll::EnumerationDyn<fplll::FP_NR<dpe_struct [1]> >::process_solution(double) in enumerate.o
      fplll::EnumerationDyn<fplll::FP_NR<long double> >::process_solution(double) in enumerate.o
      fplll::EnumerationDyn<fplll::FP_NR<double> >::process_solution(double) in enumerate.o
      fplll::FastEvaluator<fplll::FP_NR<__mpfr_struct [1]> >::eval_sol(std::vector<fplll::FP_NR<__mpfr_struct [1]>, std::allocator<fplll::FP_NR<__mpfr_struct [1]> > > const&, double const&, double&) in enumerate.o
      fplll::FastEvaluator<fplll::FP_NR<dpe_struct [1]> >::eval_sol(std::vector<fplll::FP_NR<dpe_struct [1]>, std::allocator<fplll::FP_NR<dpe_struct [1]> > > const&, double const&, double&) in enumerate.o
      ...
  "std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      ...
  "std::domain_error::domain_error(char const*)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::const_iterator::operator==(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::const_iterator const&) const in bkz_param.o
  "std::domain_error::domain_error(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.285] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.286] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.287] in bkz_param.o
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&>(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&) in bkz_param.o
      unsigned long nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::get_impl<unsigned long, 0>(unsigned long*) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
      ...
  "std::out_of_range::out_of_range(char const*)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::const_iterator::operator*() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
  "std::runtime_error::runtime_error(char const*)", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) [clone .constprop.727] in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_preprocessing(int, int, fplll::BKZParam const&) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) [clone .constprop.728] in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::svp_reduction(int, int, fplll::BKZParam const&, bool) [clone .constprop.731] in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::svp_preprocessing(int, int, fplll::BKZParam const&) in bkz.o
      ...
  "std::basic_ifstream<char, std::char_traits<char> >::basic_ifstream(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::_Ios_Openmode)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
  "std::invalid_argument::invalid_argument(char const*)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      fplll::Pruner<fplll::FP_NR<double> >::measure_metric(std::vector<double, std::allocator<double> > const&) in pruner.o
      fplll::Pruner<fplll::FP_NR<double> >::measure_metric(std::array<fplll::FP_NR<double>, 1023ul> const&) in pruner.o
      fplll::Pruner<fplll::FP_NR<double> >::repeated_enum_cost(std::array<fplll::FP_NR<double>, 1023ul> const&) in pruner.o
      fplll::Pruner<fplll::FP_NR<double> >::greedy(std::array<fplll::FP_NR<double>, 1023ul>&) in pruner.o
      fplll::FP_NR<double> fplll::svp_probability<fplll::FP_NR<double> >(fplll::Pruning const&) in pruner.o
      fplll::FP_NR<double> fplll::svp_probability<fplll::FP_NR<double> >(std::vector<double, std::allocator<double> > const&) in pruner.o
      ...
  "std::invalid_argument::invalid_argument(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::expect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace(unsigned long, unsigned long, char const*, unsigned long)", referenced from:
      fplll::load_strategies_json(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&) in bkz_param.o
      void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&>(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const&) in bkz_param.o
      unsigned long nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::get_impl<unsigned long, 0>(unsigned long*) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::expect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
      ...
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_replace_aux(unsigned long, unsigned long, unsigned long, char)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_erase(unsigned long, unsigned long)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long)", referenced from:
      fplll::strategy_full_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::get_string() const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::expect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
      ...
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long)", referenced from:
      void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.189] in bkz_param.o
      fplll::default_strategy_path[abi:cxx11]()  in bkz_param.o
      fplll::strategy_full_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      fplll::default_strategy[abi:cxx11]()  in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.285] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator> const& nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::operator[]<char const>(char const*) const [clone .constprop.287] in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::unexpect(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::token_type) const in bkz_param.o
      ...
  "std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()", referenced from:
      fplll::default_strategy_path[abi:cxx11]()  in bkz_param.o
      fplll::strategy_full_path(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) in bkz_param.o
      fplll::default_strategy[abi:cxx11]()  in bkz_param.o
  "std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::basic_ostringstream(std::_Ios_Openmode)", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      ...
  "std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >::~basic_ostringstream()", referenced from:
      fplll::BKZReduction<fplll::FP_NR<double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<double> >::bkz() in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::tour(int, int&, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::slide_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      fplll::BKZReduction<fplll::FP_NR<long double> >::sd_tour(int, fplll::BKZParam const&, int, int) in bkz.o
      ...
  "std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)", referenced from:
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_2reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_2reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_3reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_3reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_4reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_4reduce(ListPoint<long>*) in sieve_gauss.o
  "std::__detail::_List_node_base::_M_unhook()", referenced from:
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_2reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_3reduce_2reduce(ListPoint<__mpz_struct [1]>*, std::_List_iterator<ListPoint<__mpz_struct [1]>*>&) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_3reduce_2reduce(ListPoint<long>*, std::_List_iterator<ListPoint<long>*>&) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_2reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_4reduce_3reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<long, fplll::FP_NR<double> >::update_p_3reduce(ListPoint<long>*) in sieve_gauss.o
      GaussSieve<__mpz_struct [1], fplll::FP_NR<double> >::update_p_3reduce(ListPoint<__mpz_struct [1]>*) in sieve_gauss.o
      ...
  "std::__throw_bad_function_call()", referenced from:
      fplll::ExternalEnumeration<fplll::FP_NR<__mpfr_struct [1]> >::enumerate(int, int, fplll::FP_NR<__mpfr_struct [1]>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      fplll::ExternalEnumeration<fplll::FP_NR<double> >::enumerate(int, int, fplll::FP_NR<double>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      fplll::ExternalEnumeration<fplll::FP_NR<long double> >::enumerate(int, int, fplll::FP_NR<long double>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      fplll::ExternalEnumeration<fplll::FP_NR<dpe_struct [1]> >::enumerate(int, int, fplll::FP_NR<dpe_struct [1]>&, long, std::vector<double, std::allocator<double> > const&, bool) in enumerate_ext.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::parser::parse_internal(bool) in bkz_param.o
  "std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char)", referenced from:
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::fill_line_buffer() in bkz_param.o
      nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator>::lexer::scan() in bkz_param.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status

comment:286 follow-up: Changed 17 months ago by fbissey

Which version of OS X are you running?

comment:287 Changed 17 months ago by fbissey

Successfully built with gcc-7.1 from hpc.sourceforge.net.

comment:288 in reply to: ↑ 286 Changed 17 months ago by jhpalmieri

Replying to fbissey:

Which version of OS X are you running?

Sierra (10.12.5).

Successfully built with gcc-7.1 from hpc.sourceforge.net.

Me too.

comment:289 Changed 17 months ago by git

  • Commit changed from 777a393a2344773c293595d00b469b5bb4115b70 to fb86df818df776b120072e2658c3298e6165953b

Branch pushed to git repo; I updated commit sha1. New commits:

d76da7bMerge branch 'develop' into compiler
a50d44fbump config tarball
31ccf12Merge branch 'compiler' into clang-progress
fb86df8update configure tarball

comment:290 Changed 17 months ago by fbissey

  • Description modified (diff)

comment:291 Changed 17 months ago by fbissey

@jhpalmieri your problem with 5.4.0 from homebrew is very strange. I don't any of the tricks in gcc's spkg-install deal with something like that. It is like it is trying to use the libstdc++ from the wrong arch to link. That's utterly bizarre.

comment:292 follow-up: Changed 17 months ago by jhpalmieri

It's not homebrew, I just got a gcc tarball and essentially did ./configure; make; make install. Maybe some configure flags should be added.

comment:293 in reply to: ↑ 292 Changed 17 months ago by fbissey

Replying to jhpalmieri:

It's not homebrew, I just got a gcc tarball and essentially did ./configure; make; make install. Maybe some configure flags should be added.

Possibly. That's one of these things that are hard to assess without being over your shoulder.

comment:294 Changed 17 months ago by git

  • Commit changed from fb86df818df776b120072e2658c3298e6165953b to dfebfafc0b44e1134600008a7920c303c5a59de3

Branch pushed to git repo; I updated commit sha1. New commits:

dfebfafsimplify spkg-install for gfortran a bit

comment:295 follow-up: Changed 17 months ago by kcrisman

Testing with a from-scratch git clone on OS X 10.11.6, Xcode 7.3.1. So far, so good ...

checking for C compiler vendor... clang
checking for clang... clang

and I'm not seeing the usual message about building a small mpir to build gcc.

comment:296 follow-up: Changed 17 months ago by kcrisman

BUT

Found local metadata for gfortran-5.4.0
Attempting to download package gcc-5.4.0.tar.bz2 from mirrors
http://files.sagemath.org/spkg/upstream/gfortran/gcc-5.4.0.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//files.sagemath.org/spkg/ups$
http://mirrors-usa.go-parts.com/sage/sagemath/spkg/upstream/gfortran/gcc-5.4.0.$
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//mirrors-usa.go-parts.com/sa$
http://mirror.yandex.ru/mirrors/sage.math.washington.edu/spkg/upstream/gfortran$
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]

Etc.

And this makes sense, because http://files.sagemath.org/spkg/upstream/ doesn't have a gfortran directory. So I had to download it "by hand" from files.sagemath.org. Something along these lines would have to be fixed eventually. But for testing purposes at the least you should point this out in your instructions.

For now, that seemed to do it. Though

[gfortran-5.4.0] The following languages will be built: c,fortran

Does it need to build c?

comment:297 in reply to: ↑ 296 Changed 17 months ago by dimpase

Replying to kcrisman:

For now, that seemed to do it. Though

[gfortran-5.4.0] The following languages will be built: c,fortran

Does it need to build c?

IMHO it cannot at the moment build Fortran only, it has to be C as well.

comment:298 Changed 17 months ago by jhpalmieri

By the way, I reinstalled my own gcc-5.4.0, and now Sage builds and all tests pass. So it looks like the problems were on my end.

comment:299 Changed 17 months ago by kcrisman

R was apparently successfully built in my case, so that is a very good sign.

comment:300 in reply to: ↑ 295 ; follow-up: Changed 17 months ago by kcrisman

Testing with a from-scratch git clone on OS X 10.11.6, Xcode 7.3.1. So far, so good ...

Sage successfully built, modulo needing to download those two files. Currently building doc but that should be unrelated. Shall I run doctests?

comment:301 in reply to: ↑ 300 Changed 17 months ago by fbissey

Replying to kcrisman:

Testing with a from-scratch git clone on OS X 10.11.6, Xcode 7.3.1. So far, so good ...

Sage successfully built, modulo needing to download those two files. Currently building doc but that should be unrelated. Shall I run doctests?

yes please! Thanks for the feedback.

I had always assumed people would start from a tarball so yes I will add instructions shortly. If we go that route the gfortran folder in the upstream repo should link to the gcc one.

@ jhpalmieri That an external gcc-5.4.0 works is great news.

Considering some feedback on #22646 I may have to introduce some versioning check. xcode 7 apparently works, I don't know if someone can try xcode 6. But xcode 7 is probably a good lower bound on OS X. Now apparently clang 3.7 on fedora 23 is quite broken so on linux I am thinking of putting the lower bound at 3.8. But install details may vary from distro to distro - incredibly I think clang is too modular for its own good.

comment:302 Changed 17 months ago by fbissey

  • Description modified (diff)

comment:303 Changed 17 months ago by dimpase

Could we enable freebsd on this branch?

(this link is to the corresponding trivial commit from #22679)

It looks as if it's the only thing needed on top of the branch here to build on freebsd (well, still with a bit of manual intervention).

comment:304 Changed 17 months ago by kcrisman

  • Description modified (diff)

comment:305 follow-up: Changed 17 months ago by kcrisman

No errors but an awful lot of timeouts, though that said I haven't run doctests for ages on this computer. I'll try again with SAGE_TIMEOUT=1000 but I don't expect much.

comment:306 in reply to: ↑ 305 ; follow-up: Changed 17 months ago by fbissey

Replying to kcrisman:

No errors but an awful lot of timeouts, though that said I haven't run doctests for ages on this computer. I'll try again with SAGE_TIMEOUT=1000 but I don't expect much.

That's unusual. Is the machine overloaded? Of course that could be down to one single critical element with a fault causing the timeouts.

comment:307 in reply to: ↑ 306 Changed 17 months ago by kcrisman

No errors but an awful lot of timeouts, though that said I haven't run doctests for ages on this computer. I'll try again with SAGE_TIMEOUT=1000 but I don't expect much.

That's unusual. Is the machine overloaded? Of course that could be down to one single critical element with a fault causing the timeouts.

I think it just couldn't handle parallel testing of some files - everything that didn't pass was over 100s when done individual and was fine. The only thing that didn't actually pass at some point was

sage -t src/sage/repl/configuration.py
**********************************************************************
File "src/sage/repl/configuration.py", line 14, in sage.repl.configuration
Failed example:
    'In [1]: [False, True]' in output
Expected:
    True
Got:
    False
**********************************************************************
1 item had failures:
   1 of   5 in sage.repl.configuration
    [18 tests, 1 failure, 31.96 s]

but this passed three times in a row when done individually so perhaps some obscure race condition. I say that this means we can use clang on my box, anyway.

comment:308 Changed 17 months ago by dimpase

this works nicely on FreeBSD, with a small patch on top of this branch, see #22679. (few failing doctests, a bit more than on OSX).

comment:309 follow-up: Changed 17 months ago by dimpase

On Gentoo Linux with clang (4.0.0), there are C++ problems, no matter what -stdlib= is.

  • -stdlib=libstdc++: in giac there are problems with hash_map and unordered_map - none work. Not sure it's gentoo or giac bug, or both... While the rest of the packages appear to build, this one does not look easy to tackle without the author. Another reason to stick to libc++, for the time being at least.
  • -stdlib=libc++: some C++ problems (do not look to hard to fix) in eclib. More to come (still compiling more packages, to see what else is iffy)

comment:310 in reply to: ↑ 309 Changed 17 months ago by fbissey

Replying to dimpase:

On Gentoo Linux with clang (4.0.0), there are C++ problems, no matter what -stdlib= is.

  • -stdlib=libstdc++: in giac there are problems with hash_map and unordered_map - none work. Not sure it's gentoo or giac bug, or both... While the rest of the packages appear to build, this one does not look easy to tackle without the author. Another reason to stick to libc++, for the time being at least.
  • -stdlib=libc++: some C++ problems (do not look to hard to fix) in eclib. More to come (still compiling more packages, to see what else is iffy)

Gentoo but I am still on clang-3.9.1.

There are indeed some issues with eclib. Apart from the tolerance issue one the test executable segfault for an unclear reason in a C++ header from clang's libstdc++. The issue doesn't appear to exist on OS X - where xcode is based on clang 3.8. eclib could gain to completely switch to c++11. It wouldn't fix all the issues but it would come out cleaner. The issues in eclib don't prevent sage test suite to fully pass.

giac, yes lots of warnings but apart from the funny test where the order of the solution flip between gcc and clang everything passes. I must say that I already have a bad taste in my mouth from hash_map/unordered_map in brial. I think hash_map should get the cold shoulder.

Now for something a little bit different. The branch in its current form will let you try to compile sage with the intel compiler (C/C++ not fortran). So far I only needed two fixes aside from the one already needed for clang. sqlite (because icc fakes gcc too much and sqlite didn't think of it) and openblas (known issue fixed in master but that missed the 0.2.19 release).

comment:311 follow-up: Changed 17 months ago by dimpase

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors. How does one submit patches for giac?

comment:312 in reply to: ↑ 311 ; follow-up: Changed 17 months ago by fbissey

Replying to dimpase:

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors.

OK, makes sense. We probably want to switch most of these to std=c++11 anyway. I am writing a pull request for eclib to change its default and drop the stuff that c++11 doesn't like (keyword "register").

How does one submit patches for giac?

You have to raise an issue on their forum at http://xcas.e.ujf-grenoble.fr/XCAS , note that me and debian are already queuing for some other changes related to gui and the use of X libraries.

comment:313 in reply to: ↑ 312 ; follow-up: Changed 17 months ago by dimpase

Replying to fbissey:

Replying to dimpase:

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors.

Unintentionally, I built with -stdlib=libstdc++; apparently in my configuration I need an explicit -stdlib=libc++ in CXXFLAGS.

Anyhow, ptestlong fails just one doctest, due to numerical noise:

File "src/sage/libs/lcalc/lcalc_Lfunction.pyx", line 188, in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
Failed example:
    L.hardy_z_function(0)
Expected:
    0.7939675904771...
Got:
    0.793967590477206 - 4.03312243040125e-17*I

I'd say this configuration is good to go. I'll test with -stdlib=libc++, too. I guess some GNUisms might pop up...

OK, makes sense. We probably want to switch most of these to std=c++11 anyway. I am writing a pull request for eclib to change its default and drop the stuff that c++11 doesn't like (keyword "register").

Yes, this is a good idea.

How does one submit patches for giac?

You have to raise an issue on their forum at http://xcas.e.ujf-grenoble.fr/XCAS , note that me and debian are already queuing for some other changes related to gui and the use of X libraries.

It seems I need to jump through a French-only hoop to subscribe to that forum :-)

comment:314 in reply to: ↑ 313 Changed 17 months ago by fbissey

Replying to dimpase:

Replying to fbissey:

Replying to dimpase:

On Gentoo Linux with clang 4.0.1 and libc++, things are much better than on 4.0.0, a successful, almost automatic, build: one needs CXXFLAGS="-std=c++11" for giac, ntl, and eclib, but no such flag for lcalc; also, giac needs a straightforward patch (several static_cast<>())

Without that CXXFLAGS some old gcc headers get picked up, leading to lots of C++ errors.

Unintentionally, I built with -stdlib=libstdc++; apparently in my configuration I need an explicit -stdlib=libc++ in CXXFLAGS.

Anyhow, ptestlong fails just one doctest, due to numerical noise:

File "src/sage/libs/lcalc/lcalc_Lfunction.pyx", line 188, in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
Failed example:
    L.hardy_z_function(0)
Expected:
    0.7939675904771...
Got:
    0.793967590477206 - 4.03312243040125e-17*I

That's funny.

fbissey@moonloop ~/sandbox/git-fork/sage-icc $ ./sage -t --long src/sage/libs/lcalc/lcalc_Lfunction.pyx 
too many failed tests, not using stored timings
Running doctests with ID 2017-06-29-20-00-14-17cf5c10.
Git branch: clang-progress
Using --optional=mpir,python2,sage
Doctesting 1 file.
sage -t --long src/sage/libs/lcalc/lcalc_Lfunction.pyx
**********************************************************************
File "src/sage/libs/lcalc/lcalc_Lfunction.pyx", line 188, in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
Failed example:
    L.hardy_z_function(0)
Expected:
    0.7939675904771...
Got:
    0.793967590477208 - 4.03312243040124e-17*I
**********************************************************************
1 item had failures:
   1 of  15 in sage.libs.lcalc.lcalc_Lfunction.Lfunction.hardy_z_function
    [103 tests, 1 failure, 1.10 s]

Yes, this is my icc build. I think it deserves a separate numerical noise ticket.

I'd say this configuration is good to go. I'll test with -stdlib=libc++, too. I guess some GNUisms might pop up...

OK, makes sense. We probably want to switch most of these to std=c++11 anyway. I am writing a pull request for eclib to change its default and drop the stuff that c++11 doesn't like (keyword "register").

Yes, this is a good idea.

I should surprise John with that PR soon.

How does one submit patches for giac?

You have to raise an issue on their forum at http://xcas.e.ujf-grenoble.fr/XCAS , note that me and debian are already queuing for some other changes related to gui and the use of X libraries.

It seems I need to jump through a French-only hoop to subscribe to that forum :-)

Was it all French? I can't remember :)

comment:315 follow-up: Changed 17 months ago by dimpase

with clang 4.0.1 and -stdlib=libc++ -std=c++11 for eclib to build I need

diff --git a/libsrc/eclib/interface.h b/libsrc/eclib/interface.h
index b8eff26..20c9c98 100644
--- a/libsrc/eclib/interface.h
+++ b/libsrc/eclib/interface.h
@@ -159,10 +159,7 @@ inline int is_approx_zero(const RR& x)
   // cout<<"is_approx_zero() returns "<<(x.mantissa()<power2_ZZ(-n))<<endl;
   return abs(x.mantissa())<power2_ZZ(-n);
 }
-} // namespace NTL
 #ifdef _LIBCPP_VERSION
-namespace std {
-inline namespace __1 {
 inline bool isinf(const RR& z) {
   return false;
 }
@@ -182,9 +179,8 @@ inline RR fmax(const RR& x, const RR& y)
 {
   return x < y ? y : x;
 }
-}
-}
 #endif
+}
 
 #include <complex>
 typedef complex<RR> CC;

which basically moves isinf, insnan, etc to the namespace NTL. Does this break clang on OSX?

comment:316 in reply to: ↑ 315 Changed 17 months ago by fbissey

Replying to dimpase:

which basically moves isinf, insnan, etc to the namespace NTL. Does this break clang on OSX?

I will try and see what it does here.

comment:317 Changed 17 months ago by fbissey

Makes no difference whatsoever on OS X. But it reminded me that I had the same weird segfault on OS X as I have on linux with clang 3.9.1 (in the testsuite).

comment:318 Changed 17 months ago by git

  • Commit changed from dfebfafc0b44e1134600008a7920c303c5a59de3 to 7a9ab6b0cbc9625e1b5df0a29536803e8d7bfab7

Branch pushed to git repo; I updated commit sha1. New commits:

5434c70Merge branch 'develop' into compiler
f565739bump config tarball
53f9c4dMerge branch 'compiler' into clang-progress
7a9ab6bchecksum for this branch's configure tarball

comment:319 Changed 17 months ago by fbissey

  • Description modified (diff)

comment:320 follow-up: Changed 17 months ago by fbissey

Tried clang-4.0.1 on linux still the same stuff happening during the testsuite. Will see what happens on OS X.

The patch for eclib will need its own ticket, with a matching PR upstream and needs testing with gcc.

The lcalc doctest should also get its own ticket.

comment:321 in reply to: ↑ 320 ; follow-up: Changed 17 months ago by dimpase

Replying to fbissey:

Tried clang-4.0.1 on linux still the same stuff happening during the testsuite. Will see what happens on OS X.

The patch for eclib will need its own ticket, with a matching PR upstream and needs testing with gcc.

The lcalc doctest should also get its own ticket.

On Linux clang 4.0.1 with libc++ gives a weird linking error on gfan (another old C++ worry).

It took me a while to notice that CXXFLAGS is not used during compilation of C++. (and in my setup -stdlib=libc++ is not the default, the default is libstdc++, so CXXFLAGS are really needed!) It's an error in gfan's makefile that somehow went unnoticed. The following (a patch to Sage's patch :-)) fixes it:

diff --git a/build/pkgs/gfan/patches/Makefile.patch b/build/pkgs/gfan/patches/Makefile.patch
index 648fa56902..2c4e1d2d3f 100644
--- a/build/pkgs/gfan/patches/Makefile.patch
+++ b/build/pkgs/gfan/patches/Makefile.patch
@@ -60,7 +60,7 @@ index 0892969..c626eb9 100644
        $(CXX) -c $<
  .cpp.o:
 -      $(CXX) $(CFLAGS) -c $<
-+      $(CXX) $(CFLAGS_LOCAL) $(CPPFLAGS) -c $<
++      $(CXX) $(CXXFLAGS_LOCAL) $(CPPFLAGS) -c $<
  .C.o:
        $(CXX) -c $<
  # wget http://ftp.sunet.se/pub/gnu/gmp/gmp-4.2.2.tar.gz

Should we have a separate ticket for this?

comment:322 in reply to: ↑ 321 Changed 17 months ago by fbissey

Replying to dimpase:

Should we have a separate ticket for this?

Yes.

comment:323 follow-up: Changed 17 months ago by jdemeyer

Bikeshed: I prefer configure messages to be short. 3 lines just to say that gfortran needs to be installed is too much:

AC_MSG_NOTICE([GFORTRAN will be built.])
AC_MSG_NOTICE([Your Fortran compiler isn't GCC (GNU Fortran) or])
AC_MSG_NOTICE([you don't have one.])

comment:324 follow-up: Changed 17 months ago by jdemeyer

Would it be possible to treat gfortran as a completely normal package? That means, do not build it during all-toolchain, but just add it explicitly as dependency to the packages that need it. That would speed up compilation since gfortran could be built in parallel with other packages (the toolchain is built serially, so it should contain as few packages as possible).

comment:325 Changed 17 months ago by jdemeyer

Please do create new tickets for further issues regarding specific packages. This ticket is nicely self-contained, dealing only with the basic Sage configure/build system.

comment:326 in reply to: ↑ 324 Changed 17 months ago by dimpase

Replying to jdemeyer:

Would it be possible to treat gfortran as a completely normal package? That means, do not build it during all-toolchain, but just add it explicitly as dependency to the packages that need it. That would speed up compilation since gfortran could be built in parallel with other packages (the toolchain is built serially, so it should contain as few packages as possible).

Sure, why not - except that then all the packages that need it must get the corresponding dependency, on the package gfortran. (it's relatively few: numpy, scipy, R, anything esle?)

comment:327 in reply to: ↑ 323 Changed 17 months ago by fbissey

Replying to jdemeyer:

Bikeshed: I prefer configure messages to be short. 3 lines just to say that gfortran needs to be installed is too much:

AC_MSG_NOTICE([GFORTRAN will be built.])
AC_MSG_NOTICE([Your Fortran compiler isn't GCC (GNU Fortran) or])
AC_MSG_NOTICE([you don't have one.])

Fair enough. I'll try to be more terse. The previous message was annoying me as potentially misleading: "Your fortran compiler is not GCC..." it creates the perception you have a fortran compiler even when you have none.

Dima has a fair point about dependency on gfortran. But the only dependency needed is for blas since all the package needing fortran need it for lapack.

Last edited 17 months ago by fbissey (previous) (diff)

comment:328 Changed 17 months ago by kcrisman

Dima has a fair point about dependency on gfortran. But the only dependency needed is for blas since all the package needing fortran need it for lapack.

Oh, R doesn't need it directly? Certainly some R packages do need fortran directly, though of course supporting every R package in existence might be tricky.

comment:329 Changed 17 months ago by jdemeyer

To be safe, we could (and should) add gfortran explicitly as a dependency of R. I guess the main point of François was that it's not a problem if we forget one package which needs Fortran because of BLAS.

comment:330 Changed 17 months ago by fbissey

R depends on lapack, although you are right some packages depends on fortran directly. The same is true for scipy, it depends on lapack and uses some fortran code directly. I am fairly sure numpy only depends on lapack/blas. Some optional packages would need to be checked (coinor uses lapack).

If we include the external dependencies required by cvxopt 1.1.9 some of them may depend directly on fortran.

comment:331 follow-up: Changed 17 months ago by fbissey

A practical question about putting gfortran in dependencies. If you use a system gfortran, a logs/pkg/gfortran-x.log is not produced. Same as with gcc. How does the dependency system knows that gfortran has been installed? I am guessing I should look at yasm for an example.

comment:332 Changed 17 months ago by fbissey

Somehow it must work packages that need to depend on (g)fortran - add any packages that missing to the list:

  • openblas
  • atlas
  • R
  • scipy
  • numpy (linking with lapack means that numpy needs to at least know about your fortran compiler apparently)

On the top of my head none of the other packages in standard, optional or experimental needs fortran. But I am unfamiliar with some of those packages in optional and experimental.

comment:333 Changed 17 months ago by git

  • Commit changed from 7a9ab6b0cbc9625e1b5df0a29536803e8d7bfab7 to 5326571ec55c1d533f51bb85c0237483785a3929

Branch pushed to git repo; I updated commit sha1. New commits:

5326571make gfortran a regular dependency. Simplify message in configure.

comment:334 Changed 17 months ago by fbissey

  • Milestone changed from sage-8.0 to sage-8.1

comment:335 Changed 17 months ago by fbissey

  • Dependencies changed from #22799 #22646 to #22799 #22646 #23354

comment:336 Changed 17 months ago by jdemeyer

  • Dependencies changed from #22799 #22646 #23354 to #22646

See 325

comment:337 in reply to: ↑ 331 Changed 17 months ago by jdemeyer

Replying to fbissey:

A practical question about putting gfortran in dependencies. If you use a system gfortran, a logs/pkg/gfortran-x.log is not produced. Same as with gcc. How does the dependency system knows that gfortran has been installed? I am guessing I should look at yasm for an example.

The configure variable need_to_install_gfortran adjusts the generated build/make/Makefile. And indeed yasm is a good example.

comment:338 Changed 17 months ago by git

  • Commit changed from 5326571ec55c1d533f51bb85c0237483785a3929 to e89c07c6be61c8abd9c3653e3cc6cfde7535fe8b

Branch pushed to git repo; I updated commit sha1. New commits:

f4d6748Change the setting of compilers in the order: environment, sage installed compilers, configured compilers.
eb6f027fix objective compiler detection. Don't try to negate using the macro. Bad syntax that no one seem to have noticed.
4c41390Regenerate configure tarball
38273c6Remove bits specific to outdated xcode toolchain
ab61da6update configure tarball
5820812Merge branch 'develop' into compiler and bump configure tarball
e89c07cMerge branch 'compiler' into clang-progress and update configure tarball

comment:339 Changed 17 months ago by fbissey

  • Description modified (diff)

comment:340 Changed 17 months ago by dimpase

I noticed that the optional package boost does not build ob FreeBSD, and gets built with gcc on Linux/clang/libc++ even if CC and CXX are specified.

It seems we need to fix spkg-install there to use --toolset $CC option. As well, passing -std=... has to be done explicitly, it seems; does this mean that the toplevel Sage configure also needs something like CXXLIB= option.

comment:341 Changed 17 months ago by fbissey

Good catch, I'll have a look later today. I don't like the idea of adding CXXLIB that's very specific. But we can sort out some details later. boost has a, shall we say, very peculiar build system, only used by boost. The boost dev would you to use it for your project involving boost but it doesn't seem to get much traction (thanks goodness).

comment:342 Changed 17 months ago by fbissey

Hum... that will be interesting, it will need its own ticket of course. From bootstrap.sh

# Determine the toolset, if not already decided
if test "x$TOOLSET" = x; then
  guessed_toolset=`$my_dir/tools/build/src/engine/build.sh --guess-toolset`
  case $guessed_toolset in
    acc | darwin | gcc | como | mipspro | pathscale | pgi | qcc | vacpp )
    TOOLSET=$guessed_toolset
    ;;
    
    intel-* )
    TOOLSET=intel
    ;;
    
    mingw )
    TOOLSET=gcc
    ;;
    
    sun* )
    TOOLSET=sun
    ;;
    
    * )
    # Not supported by Boost.Build
    ;;
  esac
fi

clang is not a defined toolset, may be we'd end up with darwin (no), it should probably piggy back on gcc. May be we need an update to boost. We could possibly leave the toolset as gcc for now and use a user-config.jam file with our settings.

It must already do funny things on OS X since in the absence of toolset, darwin is used and probably default to clang (unless they are counting on gcc to be clang, it is not impossible considering the rest of the mess).

The build system is complex and dealing with it in a systematic way for other vendor (say intel) is difficult.

Last edited 17 months ago by fbissey (previous) (diff)

comment:343 Changed 16 months ago by git

  • Commit changed from e89c07c6be61c8abd9c3653e3cc6cfde7535fe8b to d20a8e70112ae1a965f039bf6266a6f83427143f

Branch pushed to git repo; I updated commit sha1. New commits:

79d0963Remove compiler selecting logic from sage-env-config and put it back in sage-env.
d20a8e7Merge branch 'compiler' into clang-progress

comment:344 Changed 16 months ago by fbissey

OK I am wrong for clang support. It is only guessed for freebsd - and then according to what I quoted it falls in the unsupported category, but that doesn't stop anything. And provided I setup things right (which I hadn't before) it builds on linux with toolset=clang.

So on looking closer at the current situation (i.e. without this ticket), on OS X we probably currently build with clang (cc seem to be used) so it is probably on shaky ground with the rest of sage right now (c++ runtime incompatibilities).

comment:345 Changed 16 months ago by git

  • Commit changed from d20a8e70112ae1a965f039bf6266a6f83427143f to 40bd88abf971c3bc2a8d1f78c75092156e581780

Branch pushed to git repo; I updated commit sha1. New commits:

1015a1fMerge branch 'master' into compiler and configure tarball bump
40bd88aMerge branch 'compiler' into clang-progress

comment:346 Changed 16 months ago by fbissey

  • Description modified (diff)

comment:347 Changed 16 months ago by git

  • Commit changed from 40bd88abf971c3bc2a8d1f78c75092156e581780 to eaf6473d4bfbcb0dec5588dd8d3b8374b5c7b2d9

Branch pushed to git repo; I updated commit sha1. New commits:

c4fce28fix missing "$" in sage-env
bbee7b8Merge branch 'develop' into compiler and bump configure tarball
eaf6473Merge branch 'compiler' into clang-progress and regenerate configure tarball

comment:348 Changed 16 months ago by fbissey

  • Description modified (diff)

Note that as of sage 8.1.beta0 this ticket can be tested out of the box on linux.

comment:349 Changed 16 months ago by dimpase

there is a mismatch with the value in checksums.ini

$ md5sum configure-232.tar.gz 
4aa680b6dfc4c372872a1a5ee4ec781b  configure-232.tar.gz

vs in checksums.ini

md5=0ab186b7307444b0f2e7f47413eb5ba2

comment:350 follow-up: Changed 16 months ago by fbissey

  • Description modified (diff)

Strange I have the right checksum. I replaced the link in the description as if you cut and paste the address that was there you'll download a web page instead of the tarball. I am wondering if that's you problem. It should be a direct link now.

comment:351 in reply to: ↑ 350 Changed 16 months ago by dimpase

Replying to fbissey:

Strange I have the right checksum. I replaced the link in the description as if you cut and paste the address that was there you'll download a web page instead of the tarball. I am wondering if that's you problem.

Oh, indeed, it's some http weirdness that creates a file named configure-232.tar.gz but it is actually not a gzipped tarfile...

Trying the branch on Linux/clang now.

comment:352 Changed 16 months ago by dimpase

OK, this builds and seems to work on Gentoo/clang/libc++, if I add CXXFLAGS="-stdlib=libc++. Without the latter (i.e. with the default libstdc++ instead) it does not work with giac and probably more.

comment:353 Changed 16 months ago by dimpase

One has however fix the problem that libgiac pulls in libstdc++.so as well as libc++.so, and this leads to giac basically being non-functional. But this is for another ticket.

comment:354 Changed 16 months ago by jhpalmieri

By the way, I've been trying this on OS X with each new Sage release, and it's worked well each time. (This is with my own gfortran installed, by the way.)

Last edited 16 months ago by jhpalmieri (previous) (diff)

comment:355 Changed 16 months ago by git

  • Commit changed from eaf6473d4bfbcb0dec5588dd8d3b8374b5c7b2d9 to 6f17c0281d618adfe7f8a79b3091c1dd3fb18ce6

Branch pushed to git repo; I updated commit sha1. New commits:

e485a75Merge branch 'develop' into compiler and bump configure tarball
6f17c02Merge branch 'compiler' into clang-progress

Changed 16 months ago by fbissey

comment:356 Changed 16 months ago by fbissey

  • Description modified (diff)

comment:357 Changed 15 months ago by git

  • Commit changed from 6f17c0281d618adfe7f8a79b3091c1dd3fb18ce6 to ceb3bf5d3274fd1fa6d8b917e4fa232ef264170e

Branch pushed to git repo; I updated commit sha1. New commits:

ceb3bf5conform to new packaging in place since 8.1.beta1

comment:358 Changed 15 months ago by dimpase

Could we have a rebase here?

comment:359 Changed 15 months ago by fbissey

How urgent is it? The main reason for the need to rebase is that I keep providing a configure tarball which is updated every beta (I think). Otherwise the branch wouldn't need rebasing so often.

comment:360 Changed 15 months ago by dimpase

Perhaps it makes sense to require autoconf instead, for the purposes of this ticket going forward at least. We can put configure tarball back once it's ready to go.

comment:361 Changed 15 months ago by fbissey

OK, it will have to be the same with #22646 on which it depends. I have stuff to put in #22646, I will proceed with the autoconf requirement from there. But I am afraid it will have to wait until I go back to work tomorrow morning in my time zone.

comment:362 Changed 15 months ago by git

  • Commit changed from ceb3bf5d3274fd1fa6d8b917e4fa232ef264170e to 07bfc5a1ae18c40a01ebd8c1169443f866c452d8

Branch pushed to git repo; I updated commit sha1. New commits:

3772ae5Merge branch 'develop' into compiler
a8c8904Add some documentation and update configure tarball
07bfc5aMerge branch 'compiler' into clang-progress

comment:363 Changed 15 months ago by fbissey

  • Description modified (diff)

OK it will still need rebasing until #22646 is merged because I still the tarball there for now. We are closing in on the review for that one so it may be necessary. But this ticket doesn't provide a tarball now.

comment:364 Changed 15 months ago by git

  • Commit changed from 07bfc5a1ae18c40a01ebd8c1169443f866c452d8 to d8cb8282b7c9d2fbc7c41f2ce2c4f910da6412f9

Branch pushed to git repo; I updated commit sha1. New commits:

99eb31fMinor edits from John Palmieri's feedback
d8cb828Merge branch 'compiler' into clang-progress

comment:365 Changed 15 months ago by fbissey

  • Description modified (diff)

comment:366 Changed 15 months ago by git

  • Commit changed from d8cb8282b7c9d2fbc7c41f2ce2c4f910da6412f9 to 0ec453b8977f9e3032d1bc334cce9013212a5728

Branch pushed to git repo; I updated commit sha1. New commits:

0ec453bMerge branch 'develop' into clang-progress

comment:367 follow-up: Changed 15 months ago by dimpase

Where is the gfortran tarball?

comment:368 in reply to: ↑ 367 ; follow-up: Changed 15 months ago by fbissey

Replying to dimpase:

Where is the gfortran tarball?

No gfortran tarball, gcc's tarball is used.

comment:369 in reply to: ↑ 368 Changed 15 months ago by dimpase

Replying to fbissey:

Replying to dimpase:

Where is the gfortran tarball?

No gfortran tarball, gcc's tarball is used.

the spkg-install of gfortran package is too dumb to know this, and without gfortran or gcc, and CC set to clang this ends up in an error downloading a non-existent tarball...

comment:370 Changed 15 months ago by fbissey

This shouldn't happen, spkg-install doesn't know the name of the tarball, package-version.txt does. Can you show me the log?

Last edited 15 months ago by fbissey (previous) (diff)

comment:371 Changed 15 months ago by dimpase

it was already mentioned here; namely, there is no gfortran/ directory on the servers:

Found local metadata for gfortran-5.4.0
Attempting to download package gcc-5.4.0.tar.bz2 from mirrors
Downloading the Sage mirror list
Searching fastest mirror
  149ms: http://files.sagemath.org/
...
    6ms: http://www.mirrorservice.org/sites/www.sagemath.org/
Fastest mirror: http://www.mirrorservice.org/sites/www.sagemath.org/
http://www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/gfortran/gcc-5.4.0.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [transfer|run:135]: [Errno 404] Not Found: '//www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/gfortran/gcc-5.4.0.tar.bz2'
...

comment:372 Changed 15 months ago by fbissey

Yes it says in the description that you have to make sure the gcc tarball is already downloaded. The release manager needs to create a link gfortran -> gcc when merging the ticket. This is also in the description.

comment:373 Changed 15 months ago by dimpase

Oops, sorry, I was too quick regarding the tarball. Anyway, in this configuration on FreeBSD one is hit by the usual platform specific complex numbers support SNAFU.

[gfortran-5.4.0] libtool: compile:  /usr/home/dima/Sage/porting/sage/local/var/tmp/sage/build/gfortran-5.4.0/gcc-build/./gcc/xgcc -B/
usr/home/dima/Sage/porting/sage/local/var/tmp/sage/build/gfortran-5.4.0/gcc-build/./gcc/ -B/usr/home/dima/Sage/porting/sage/local/x86
o -c ../../../src/libgfortran/runtime/memory.c  -fPIC -DPIC -o .libs/memory.o
[gfortran-5.4.0] In file included from ../../../src/libgfortran/libgfortran.h:50:0,
[gfortran-5.4.0]                  from ../../../src/libgfortran/fmain.c:4:
[gfortran-5.4.0] ../../../src/libgfortran/c99_protos.h:311:14: error: expected declaration specifiers or '...' before '(' token
[gfortran-5.4.0]  extern float cargf (float complex);
[gfortran-5.4.0]               ^
[gfortran-5.4.0] ../../../src/libgfortran/c99_protos.h:311:14: error: expected declaration specifiers or '...' before '(' token
[gfortran-5.4.0]  extern float cargf (float complex);
[gfortran-5.4.0]               ^
[gfortran-5.4.0] ../../../src/libgfortran/c99_protos.h:316:15: error: expected declaration specifiers or '...' before '(' token
[gfortran-5.4.0]  extern double carg (double complex);
[gfortran-5.4.0]                ^

IMHO we should abandon this particular FreeBSD configuration, it's too much trouble, as gfortran is readily available on the system.

comment:374 Changed 14 months ago by jhpalmieri

With the latest release of Xcode (version 9.0) on OS X, eclib now fails to build for me (using Sage 8.1.beta5 plus this branch). See the log file.

comment:375 Changed 14 months ago by fbissey

You are ahead of me. I am surprised by the problem in eclib, it looks like the new apple clang is based on 3.9 which I have been initially working with on linux. And everything is fine with 4.0 both on linux and OS X. I am not sure if the problem is really in eclib or ntl.

comment:376 follow-up: Changed 14 months ago by jhpalmieri

Can you reproduce the problem? (You may not want to install Xcode 9.0, of course.)

comment:377 in reply to: ↑ 376 Changed 14 months ago by fbissey

Replying to jhpalmieri:

(You may not want to install Xcode 9.0, of course.)

Too late by a couple of hours ^_^. I am a bit busy right now (and it is lunch time too) so it will have to wait.

Last edited 14 months ago by fbissey (previous) (diff)

comment:379 Changed 14 months ago by git

  • Commit changed from 0ec453b8977f9e3032d1bc334cce9013212a5728 to 40d32145ccd872f5a8164cc8186f1327d2d00b9e

Branch pushed to git repo; I updated commit sha1. New commits:

83815b2Merge branch 'develop' into clang-progress
40d3214Merge branch 'develop' into clang-progress

comment:380 follow-up: Changed 14 months ago by jdemeyer

Shouldn't gfortran be an order-only dependency (after the | instead of before)?

comment:381 Changed 14 months ago by jdemeyer

  • Status changed from needs_review to needs_work

Various comments about gfortran/spkg-install:

  1. Replace
    if [ -a "$SAGE_LOCAL"/bin/gcc ]; then
        echo >&2 "gcc appear to already be installed ... exiting"
        exit 1
    fi
    

by

if [ -x "$SAGE_LOCAL"/bin/gcc ]; then
    echo >&2 "Error: gcc is already installed"
    exit 1
fi
  1. First doing cd src and then immediately cd .. is silly: drop that.
  1. This isn't needed, the usual build dependency tracking will handle it:
    # Force re-configuration: the next time that "make" is run, we don't
    # want GFORTRAN to be built again, see Trac #19324
    touch "$SAGE_ROOT/configure"
    

comment:382 Changed 14 months ago by jdemeyer

In configure.ac, I would remove AC_MSG_NOTICE([GFORTRAN wasn't found, so it will be built.]): there is no need to mention every package that will be built.

comment:383 Changed 14 months ago by fbissey

Thanks for the comments, I haven't had good ones in some time :) About (3), I picked that bit from gcc's spkg-install - in a lot of ways gfortran is "gcc lite" - does that mean that this could be dropped from gcc's spkg-install (in a different ticket)?

comment:384 Changed 14 months ago by jdemeyer

The Force re-configuration part is related to this block in GCC:

# Force re-installation of gmp, mpir, mpfr and mpc with the GCC we just built.
cd "$SAGE_SPKG_INST"
rm -f gmp-* mpir-* mpfr-* mpc-*

So it is needed for GCC but not for GFortran.

comment:385 Changed 14 months ago by jdemeyer

  • Description modified (diff)

Relevant ticket that I just created: #23919. It disables GNU extensions while compiling the Sage library. This exposes in particular the known issue with lcalc (#23341).

comment:386 follow-up: Changed 14 months ago by jdemeyer

Speaking of which, does this ticket depend on #23341? Does lcalc build with Clang currently?

comment:387 in reply to: ↑ 386 Changed 14 months ago by dimpase

Replying to jdemeyer:

Speaking of which, does this ticket depend on #23341? Does lcalc build with Clang currently?

There are many clangs around. I think the one that does not work with lcalc is clang 4.x.y, for some ranges of x and y. clang 5.0.0 does compile the current clang, and it did work with clang 3.u.v for some u and v, as well.

And there is also an issue of the standard C++ library, libstdc++ (from GCC project) vs libc++ (from Clang project); they differ quite a bit. libstdc++ with clang seems to be a deadly mix...

Oh, and there also different runtime libs on Linux that may be used...

Last edited 14 months ago by dimpase (previous) (diff)

comment:388 Changed 14 months ago by fbissey

I and a few other people have been compiling successfully sage, including lcalc, with xcode 8 on OS X, clang-4.0 (not xcode) on OS X, clang 3.9 and 4.0 on linux (Gentoo). xcode 9 and some other platforms and clang installs have problems with eclib (fixed in github master but no new release before October John said).

comment:389 Changed 14 months ago by jhpalmieri

Regarding eclib, adding a file containing

  • libsrc/eclib/interface.h

    diff --git a/libsrc/eclib/interface.h b/libsrc/eclib/interface.h
    index b8eff26..127b48e 100644
    a b inline int is_approx_zero(const RR& x) 
    161161}
    162162} // namespace NTL
    163163#ifdef _LIBCPP_VERSION
    164 namespace std {
    165 inline namespace __1 {
     164namespace NTL {
    166165inline bool isinf(const RR& z) {
    167166  return false;
    168167}
    inline RR fmax(const RR& x, const RR& y) 
    183182  return x < y ? y : x;
    184183}
    185184}
    186 }
    187185#endif
    188186
    189187#include <complex>

to the patches directory allows it to build for me on OS X with Xcode 9.0.

comment:390 Changed 14 months ago by fbissey

I guess we can spin a ticket with that one fix to eclib but just until we have the new release. The new release will properly default to c++11 and contain a number of minor fixes for clang. Ok some of the bugs revealed by eclib's testsuite (mostly fixed) are not so minor but they are not breaking sage's testsuite.

comment:391 Changed 14 months ago by jhpalmieri

From my point of view, it depends on how long it will be before a new release of eclib. It would be nice to be able to build Sage in the meantime.

comment:392 Changed 14 months ago by fbissey

The problem is only with some clang configurations, the most important one being xcode 9 for which we'll need a gcc bump as well. But yes let's put that in a new ticket until next month when we can have a release.

comment:393 in reply to: ↑ 380 ; follow-up: Changed 14 months ago by fbissey

Replying to jdemeyer:

Shouldn't gfortran be an order-only dependency (after the | instead of before)?

Tricky. Change in gcc version may or may not be accompanied by incompatible changes in fortran runtime. I should double-check on that since a change in libgfortran does mean that things need to be rebuilt. From https://gcc.gnu.org/wiki/GFortran/News it appears all the versions from 4.9.1 are fully ABI compatible (although .mod files have changed at gcc-5 but we don't have any of those I think) so far. It is not excluded that some incompatibility will be introduced in some future release.

comment:394 Changed 14 months ago by git

  • Commit changed from 40d32145ccd872f5a8164cc8186f1327d2d00b9e to afca2b66e16e33da80a8c43fb0e8576ee08182a2

Branch pushed to git repo; I updated commit sha1. New commits:

6902166Simplify gfortran spkg-install a bit. Thanks to Jeroen Demeyer for the feedback on this.
afca2b6no need to be so verbose about gfortran.

comment:395 Changed 14 months ago by fbissey

  • Dependencies changed from #22646 to #23922

comment:396 follow-up: Changed 14 months ago by git

  • Commit changed from afca2b66e16e33da80a8c43fb0e8576ee08182a2 to 48462ead230b0740fed7b78d2fa349c4cefe8c02

Branch pushed to git repo; I updated commit sha1. New commits:

235c09eIncluding upstream PR 29 to compile with xcode 9 until we have a new release.
48462eaMerge branch 'eclib-xcode9' into clang-progress

comment:397 Changed 14 months ago by dimpase

gfortran package does not respect SAGE_INSTALL_GCC=no, while it should? Or, perhaps, there should be SAGE_INSTALL_GFORTRAN variable overriding gfortran installation

(I'm trying to build with Intel's ifort compiler, and the 1st obstacle is the need to disable gfortran building...)

comment:398 follow-up: Changed 14 months ago by fbissey

Yes the current setup doesn't allow for alternative fortran compiler at all. gfortran or nothing. So it is not just a matter of not installing, and yes it would need its own variable.

What you really need to change is remove lines 477-479 of configure.ac. These lines are triggering gfortran build if GFC is not yes. GFC identifying gnu fortran. Look for GFC.

comment:399 in reply to: ↑ 393 ; follow-up: Changed 14 months ago by jdemeyer

Replying to fbissey:

Replying to jdemeyer:

Shouldn't gfortran be an order-only dependency (after the | instead of before)?

Tricky. Change in gcc version may or may not be accompanied by incompatible changes in fortran runtime.

I see. It is not just about the compiler, but also about the runtime library gfortran.so.

comment:400 in reply to: ↑ 399 Changed 14 months ago by fbissey

Replying to jdemeyer:

I see. It is not just about the compiler, but also about the runtime library gfortran.so.

Exactly.

comment:401 in reply to: ↑ 398 Changed 14 months ago by dimpase

Replying to fbissey:

Yes the current setup doesn't allow for alternative fortran compiler at all. gfortran or nothing. So it is not just a matter of not installing, and yes it would need its own variable.

What you really need to change is remove lines 477-479 of configure.ac. These lines are triggering gfortran build if GFC is not yes. GFC identifying gnu fortran. Look for GFC.

Thanks. I have opened #23926 to track the work on using other Fortran compilers.

comment:402 follow-up: Changed 13 months ago by dimpase

Does this need #23898 on OSX with Xcode 9? I gather yes, as otherwise gfortran cannot be built.

comment:403 in reply to: ↑ 402 Changed 13 months ago by jhpalmieri

Replying to dimpase:

Does this need #23898 on OSX with Xcode 9? I gather yes, as otherwise gfortran cannot be built.

Yes, I guess it does need #23898, or at least the gfortran build fails without it. I haven't tried with it yet.

comment:404 Changed 13 months ago by jhpalmieri

Now I've tried with and without #23898, and I can confirm that an upgraded GCC is necessary to build gfortran.

comment:405 Changed 13 months ago by dimpase

  • Dependencies changed from #23922 to #23922, #23898

comment:406 in reply to: ↑ 396 Changed 13 months ago by dimpase

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

235c09eIncluding upstream PR 29 to compile with xcode 9 until we have a new release.
48462eaMerge branch 'eclib-xcode9' into clang-progress

with progress made on #23922, the PR29 eclib patch must be removed.

comment:407 follow-up: Changed 13 months ago by dimpase

Can we decouple fortran issues from this ticket, have it done in this reduced form, and open a new one to get gfortran package in shape? (E.g. for systems that have fortran, e.g. FreeBSD, or perhaps OSX with gfortran from some non-Sage source, fortran is not relevant).

comment:408 Changed 13 months ago by git

  • Commit changed from 48462ead230b0740fed7b78d2fa349c4cefe8c02 to 1c5c98541e843cb021552554825d1027c25a656b

Branch pushed to git repo; I updated commit sha1. New commits:

1c5c985Merge branch 'develop' into clang-progress

comment:409 in reply to: ↑ 407 Changed 13 months ago by fbissey

Replying to dimpase:

Can we decouple fortran issues from this ticket, have it done in this reduced form, and open a new one to get gfortran package in shape? (E.g. for systems that have fortran, e.g. FreeBSD, or perhaps OSX with gfortran from some non-Sage source, fortran is not relevant).

I am not sure I am very eager about this but this can definitely be done. If you think it can be merged quickly, I can separate the gfortran bits. I guess that will help with your "other fortran" project.

comment:410 Changed 13 months ago by git

  • Commit changed from 1c5c98541e843cb021552554825d1027c25a656b to 2f85d6cb5142fa16a21f6311c4c4bfccf9a08110

Branch pushed to git repo; I updated commit sha1. New commits:

2f85d6cMerge branch 'develop' into clang-progress

comment:411 Changed 13 months ago by fbissey

  • Status changed from needs_work to needs_review

Putting this back in needs review. It is a bit late for sage 8.1 I think, but if I think everything is in place for it to be usable and merged at the beginning of 8.2.

The biggest obstacles are the failing testsuite of eclib and giac. I should check the new giac-1.4.x series, may be the flipping behavior has been fixed - the patch for pari over 2.7 is not needed in that version (admittedly I didn't check compiling against pari 2.7, they may just have changed what they are compiling against internally and fixed accordingly).

comment:412 Changed 13 months ago by git

  • Commit changed from 2f85d6cb5142fa16a21f6311c4c4bfccf9a08110 to 1119be3910558c210243d39743e5b55eb6a52944

Branch pushed to git repo; I updated commit sha1. New commits:

1119be3Remove eclib patch. It conflict with the new merged version of eclib

comment:413 follow-up: Changed 12 months ago by dimpase

IMHO as long as this is not yet used to build Sage for distributing, we should not worry about doctests failing due to small numerical noise or similar issues (speaking about eclib and giac, that is).

comment:414 Changed 12 months ago by dimpase

  • Reviewers set to Jeroen Demeyer, John Palmieri, Dima Pasechnik
  • Status changed from needs_review to positive_review

Works for me, on FreeBSD and on OSX. I think it has been reviewed a lot already.

comment:415 in reply to: ↑ 413 Changed 12 months ago by jdemeyer

  • Status changed from positive_review to needs_review

Replying to dimpase:

we should not worry about doctests failing due to small numerical noise or similar issues

Of course we should. Doctests shouldn't fail.

In any case, I'd like to test this ticket more before merging it. So far, I have been mostly commenting on the diff without actually testing.

comment:416 follow-ups: Changed 12 months ago by kcrisman

we should not worry about doctests failing due to small numerical noise or similar issues

Of course we should. Doctests shouldn't fail.

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind. This is the 416th comment so I think if it builds, it builds. Then open a new ticket for "passes tests".

comment:417 in reply to: ↑ 416 Changed 12 months ago by dimpase

Replying to kcrisman:

we should not worry about doctests failing due to small numerical noise or similar issues

Of course we should. Doctests shouldn't fail.

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind. This is the 416th comment so I think if it builds, it builds. Then open a new ticket for "passes tests".

Yes, that's the right thing to do here, I think. We will open up a follow-up ticket once we have full OSX results (and perhaps Linux results?) for this one.

comment:418 Changed 12 months ago by jhpalmieri

Just to clarify, the issue is not failing doctests (at least for me, make ptestlong passes all tests), but Sage packages failing their test suites.

comment:419 Changed 12 months ago by fbissey

That's exactly where we are:

  • sage doctests pass
  • two standard packages (that would be checked by SAGE_CHECK=yes) are failing their testsuite
  • optional packages test suite: unknown but for a few.

Of the two standard packages failing their testsuite:

  • eclib: some results need careful checking, some may be fixed by having some tolerance in the tests.
  • giac: it looks like the problem would be solved by going to 1.4.9-x - I tried yesterday.

comment:420 Changed 12 months ago by fbissey

For the giac upgrade, #24174 is ready for review.

comment:421 Changed 12 months ago by jhpalmieri

I tried the new giac package from #24174 plus the branch from this ticket, and it passes its test suite for me, too. (This is on two separate OS X machines, one with the current OS and Xcode, one with the previous version of each.)

comment:422 Changed 12 months ago by fbissey

Only issue with giac may be the optional giacpy_sage, but I think it is already currently broken. Other than that and eclib it would be great if we could push this for 8.2.beta0.

comment:423 follow-up: Changed 12 months ago by jhpalmieri

The eclib (autocorrected to "eclair" which sounds good) issue still looks like numerical noise to me – here is my log file – but other people should check. In any case, I am happy with this. I will let Jeroen test more.

comment:424 in reply to: ↑ 423 Changed 12 months ago by fbissey

Replying to jhpalmieri:

The eclib (autocorrected to "eclair" which sounds good) issue still looks like numerical noise to me – here is my log file – but other people should check. In any case, I am happy with this. I will let Jeroen test more.

That's because you stop after the first file to "error". If you add "-k" to the make command in spkg_check you'll find some other case that are not so obvious. Still the pure numerical noise could easily be fixed by some subtle C++ formatting and should be done. See https://github.com/JohnCremona/eclib/issues/19#issuecomment-315219838 for a more complete list of errors (the tmrank one has been fixed) and I am worried about the stuff generated by telog where some of the signs are flipped.

comment:425 Changed 12 months ago by jhpalmieri

With the patch

  • build/pkgs/eclib/spkg-check

    diff --git a/build/pkgs/eclib/spkg-check b/build/pkgs/eclib/spkg-check
    index b22b96310d..0ec9150368 100644
    a b cd src 
    1111
    1212echo
    1313echo "Now running eclib's test suite..."
    14 $MAKE check
     14echo "Running $MAKE -k check"
     15$MAKE -k check
    1516if [ $? -ne 0 ]; then
    1617    echo >&2 "Error: eclib's test suite failed to pass."
    1718    exit 1

I see numerical noise plus the following one. Is this the telog error you mentioned?

59,61c59,61
< Periods: [w_1,w_2] = [(-0.51317082433770156405898300169308452073251780098017e-48,-0.63277902985143330237168013012161275230207585428624),(2.0131613909534741204308876218764326959565176646346,-0.31638951492571665118584006506080637615103792714312)]
< tau       = (0.49999999999999999999999999999999999999999999999742,3.1814603455271474113187811974948516017119213527377) (abs(tau)=3.220510818202869536114214880382970192586945735383)
< w_R = (4.0263227819069482408617752437528653919130353292697,0)	w_IR = (2.0131613909534741204308876218764326959565176646351,0.31638951492571665118584006506080637615103792714312)
---
> Periods: [w_1,w_2] = [(0,0.6327790298514333023716801301216127523020758542864),(-2.0131613909534741204308876218764326959565176646351,-0.3163895149257166511858400650608063761510379271432)]
> tau       = (-0.5,3.1814603455271474113187811974948516017119213527374) (abs(tau)=3.2205108182028695361142148803829701925869457353831)
> w_R = (4.0263227819069482408617752437528653919130353292703,0)	w_IR = (2.0131613909534741204308876218764326959565176646351,0.3163895149257166511858400650608063761510379271432)

comment:426 Changed 12 months ago by fbissey

Yes. Some of it is still numerical noise plus sign flip-flopping.

comment:427 in reply to: ↑ 416 ; follow-up: Changed 12 months ago by jdemeyer

Replying to kcrisman:

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind.

That was for supporting "new" systems for which there was previously no support. Here, we are talking about changing things for a system on which Sage already "passes tests". So going from "passes tests" to "builds" would be a regression.

comment:428 in reply to: ↑ 427 ; follow-up: Changed 12 months ago by dimpase

Replying to jdemeyer:

Replying to kcrisman:

Arguably I have little to say here, but in the past we have often had one ticket for "builds" and another for "passes tests" - Cygwin certainly comes to mind.

That was for supporting "new" systems for which there was previously no support. Here, we are talking about changing things for a system on which Sage already "passes tests". So going from "passes tests" to "builds" would be a regression.

well, arguably clang support is needed to resurrect the freebsd port, perhaps for dragonbsd support as well. so this is new.

as well, as long as clang is not the default way to build Sage on a system, one should not insist on passing all the tests on this system, right?

comment:429 follow-up: Changed 12 months ago by kcrisman

as well, as long as clang is not the default way to build Sage on a system, one should not insist on passing all the tests on this system, right?

Exactly my thinking. Of course if clang becomes default on OS X that is a very different thing.

comment:430 in reply to: ↑ 429 Changed 12 months ago by jdemeyer

Replying to kcrisman:

Of course if clang becomes default on OS X that is a very different thing.

Correct me if I'm wrong, but isn't that part of what this ticket does?

comment:431 in reply to: ↑ 428 Changed 12 months ago by jdemeyer

Replying to dimpase:

as long as clang is not the default way to build Sage on a system

Correct me again if I'm wrong, but this ticket does make Clang the default in the sense that GCC won't be installed if Clang is installed.

comment:432 Changed 12 months ago by jdemeyer

And if I'm wrong on any of the two points I just mentioned, it would be good if somebody could update the ticket description to explain precisely what this ticket does.

comment:433 Changed 12 months ago by fbissey

  • Description modified (diff)
  • Summary changed from Make Sage build with clang (3.7+) to Make Sage build with clang (3.7+) and make it the default on OS X

You are quite correct. In fact this is better/worse than that. Any compiler C/C++ compiler that is properly detected by AC_PROG_CC and AC_PROG_CXX and sets the autoconf variable GCC to yes, can and will be used. It could be limited to clang but I think it is better to open up a bit. Other compilers are not the default on "normal" systems.

So, in the absence of a working gcc/g++, if clang/clang++ is present and working, it will be used. In the case of OS X where /usr/bin/gcc is a wrapper to clang, it will be used. Unless someone overrides it.

Unless the variables CC/CXX are set, gcc will be tested first. Because of the fact on OS X, xcode installs usr/bin/gcc that means changing the default on OS X that's absolutely correct. If you hadn't given any thoughts to it, Jeroen just put your nose in it. I should add that when I took over this ticket it was quite clear to me that it was the objective because gcc keep getting broken in funny ways, which means we do things that end up having unintended consequences. That and the fact that building gcc is probably the package that take the longest to build on OS X.

I make no bones or apologies about the fact that this ticket makes clang the default on OS X.

I am updating the description and title. Also I think the ticket is not "meta" anymore.

The status is doctests pass and all standard packages' testsuite pass except for two (one with an extra ticket). Is it good enough to change the default now? My impression is that the people using OS X are quite happy with the deal but it will trigger buildbot failures unless we do something about eclib's testsuite.

comment:434 Changed 11 months ago by git

  • Commit changed from 1119be3910558c210243d39743e5b55eb6a52944 to 59910fb3f4265da8ffe8ee34376b5f96da776e71

Branch pushed to git repo; I updated commit sha1. New commits:

59910fbMerge branch 'develop' into clang-progress

comment:435 Changed 11 months ago by fbissey

eclib is a tough nut to crack because it appears the precision is not as good as the original design may have expected it to be. Error handling and propagation is a difficult business and the use of clang just uncovered it. In some of tests we have 45 digits of precision instead of the expected 50 for example. Not a big deal for most numerical applications but not good when your tests expect 50 digits to be correct. And increasing precision can cause the sign flip flop when quantities are close to (or actually should be) 0.

comment:436 follow-up: Changed 11 months ago by dimpase

Can we have an update to the current beta?

comment:437 Changed 11 months ago by git

  • Commit changed from 59910fb3f4265da8ffe8ee34376b5f96da776e71 to 632b7bc35289a12a419c691d4debd947b59f47b6

Branch pushed to git repo; I updated commit sha1. New commits:

632b7bcMerge branch 'develop' into clang-progress

comment:438 in reply to: ↑ 436 Changed 11 months ago by fbissey

Replying to dimpase:

Can we have an update to the current beta?

I guess that's what you wanted.

comment:439 Changed 11 months ago by dimpase

We have access to a 24-core OSX server now. This should be useful for this ticket :-)

comment:440 Changed 11 months ago by dimpase

with SAGE_INSTALL_GCC=no, I get

configure: error: SAGE_INSTALL_GCC is set to 'no', but a Fortran compiler is missing

without it, gcc spkg gets installed, and then gfortran spkg install fails with "gcc already installed".

comment:441 Changed 11 months ago by dimpase

specifically, in configure I get

ac_ext=${ac_fc_srcext-f}
ac_compile='$FC -c $FCFLAGS $ac_fcflags_srcext conftest.$ac_ext >&5'
ac_link='$FC -o conftest$ac_exeext $FCFLAGS $LDFLAGS $ac_fcflags_srcext conftest.$ac_ext $LIBS >&5'
ac_compiler_gnu=$ac_cv_fc_compiler_gnu

if test -z "$FC"; then

    if test x$SAGE_INSTALL_GCC = xexists; then
        true  # Do nothing if already installed
    elif test x$SAGE_INSTALL_GCC = xno; then
        as_fn_error $? "SAGE_INSTALL_GCC is set to 'no', but a Fortran compiler is missing" "$LINENO" 5
    else
        { $as_echo "$as_me:${as_lineno-$LINENO}: Installing GCC because a Fortran compiler is missing" >&5
$as_echo "$as_me: Installing GCC because a Fortran compiler is missing" >&6;}
        need_to_install_gcc=yes
    fi

else
   # see http://www.gnu.org/software/hello/manual/autoconf/Fortran-Compiler.html

but I don't understand how as_fn_error $? "SAGE_INSTALL_GCC is set to 'no', but a Fortran compiler is missing" "$LINENO" 5 is generated from configure.ac. I presume it comes from expansion of

AC_LANG(Fortran)
if test -z "$FC"; then
    need_to_install_gfortran=yes
else
   # see http://www.gnu.org/software/hello/manual/autoconf/Fortran-Compiler.html

but how?

comment:442 Changed 11 months ago by dimpase

More frustrating is that even though replacing as_fn_error ... in the previous comment with true allows the build with clang to complete, by doing

$ SAGE_INSTALL_GCC="no" ./configure CC=cc CXX=c++
$ make 

a subsequent, say, ./sage -f r starts a full build of gcc. Arrrgh, that gcc virus. I see

...
env SAGE_BUILD_TOOLCHAIN=yes make toolchain
sage-logger -p 'sage-spkg gcc-7.2.0' '/Users/dimpase/Sage/sagetrac-mirror/logs/pkgs/gcc-7.2.0.log'
[gcc-7.2.0] Found local metadata for gcc-7.2.0
[gcc-7.2.0] Using cached file /Users/dimpase/Sage/sagetrac-mirror/upstream/gcc-7.2.0.tar.xz
...

I never heard of SAGE_BUILD_TOOLCHAIN...

Last edited 11 months ago by dimpase (previous) (diff)

comment:443 Changed 11 months ago by dimpase

OK, the latter problem is that the gcc build was still loaded via configure.ac's

AS_ECHO_N(>&7 ['TOOLCHAIN = $(inst_gcc)'])

and this got through into the build system.


It looks as if either the TOOLCHAIN business is obsolete and ought to be removed, or it needs to be updated to the case of the use of clang (i.e. it would be 'TOOLCHAIN = $(inst_gfortran)'). (I don't know its purpose anyway, and it does not seem to be documented).

Last edited 11 months ago by dimpase (previous) (diff)

comment:444 Changed 11 months ago by fbissey

Sorry to have gone dark on you. I will inspect all that with a fresh mind sometime tomorrow.

comment:445 Changed 11 months ago by git

  • Commit changed from 632b7bc35289a12a419c691d4debd947b59f47b6 to 8845edbb76a84f3323cf27a2600b687ed57cf57a

Branch pushed to git repo; I updated commit sha1. New commits:

8845edbMerge branch 'develop' into clang-progress

comment:446 Changed 11 months ago by fbissey

I checked the TOOLCHAIN business and I believe it is all correct. TOOLCHAIN can be set to $(inst_gcc) only when gcc needs to be built - if you use ccache, I think it is added there too. If it is still set when gcc doesn't need to be built there is definitely a bug somewhere in the configure script. Are you sure you have run ./bootstrap before configuring?

comment:447 follow-ups: Changed 11 months ago by dimpase

I did run bootstrap -d. Not sure if the downloaded configure tarball is the correct one. I tried to build autotools our of the tree, but it complains about old m4. And I cannot build m4 - it builds, but crashes with Illegal instruction: 4. Arrgh...

How does one create the configure tarball for downloading?

comment:448 in reply to: ↑ 447 Changed 11 months ago by fbissey

Replying to dimpase:

I did run bootstrap -d. Not sure if the downloaded configure tarball is the correct one. I tried to build autotools our of the tree, but it complains about old m4. And I cannot build m4 - it builds, but crashes with Illegal instruction: 4. Arrgh...

How does one create the configure tarball for downloading?

Usually ./bootstrap -s -i. The -i increase the version number. Just autoreconf -i does the job. Not sure about the problem with the autotool spkg.

comment:449 Changed 11 months ago by jdemeyer

The git log here is insane. Allow me to squash it...

comment:450 Changed 11 months ago by jdemeyer

  • Branch changed from u/fbissey/clang-progress to u/jdemeyer/clang-progress

comment:451 Changed 11 months ago by git

  • Commit changed from 8845edbb76a84f3323cf27a2600b687ed57cf57a to d3d8317f587c63a571e14ffd0acc05d3171ea435

Branch pushed to git repo; I updated commit sha1. New commits:

d3d8317New configure tarball

comment:452 Changed 11 months ago by jdemeyer

  • Description modified (diff)

comment:453 Changed 11 months ago by jdemeyer

  • Dependencies #23922, #23898 deleted

comment:454 in reply to: ↑ 447 Changed 11 months ago by jdemeyer

Replying to dimpase:

How does one create the configure tarball for downloading?

I just did that for you :-)

comment:455 Changed 11 months ago by dimpase

OK, so configure tarball was the culprit. Once I installed autotools out of the tree, while using the system-provided m4, I was able to run ./bootstrap, and build with clang just fine.

Sage's autotools package is broken: it tries building its own m4, but it does not work:

bash-3.2$ ls -l local/bin/m4
-rwxr-xr-x  1 dimpase  staff  293092 Jan  5 14:14 local/bin/m4
bash-3.2$ ./local/bin/m4
Illegal instruction: 4

and then unsurprisingly one gets

...
sed -e 's,@''datadir''@,/Users/dimpase/Sage/sagetrac-mirror/local/autoconf-2.13.rc1/share/autoconf,g' -e 's,@''PERL''@,/usr/bin/perl,g' autoscan.pl > autoscan.tmp && chmod +x autoscan.tmp && mv autoscan.tmp autoscan
Error: Autoconf requires GNU m4 1.1 or later
make[3]: *** [autoconf.m4f] Error 1
make[2]: *** [/Users/dimpase/Sage/sagetrac-mirror/local/autoconf-2.13.rc1] Error 2

comment:456 Changed 11 months ago by jdemeyer

  • Dependencies set to #24476
  • Description modified (diff)

comment:457 Changed 11 months ago by git

  • Commit changed from d3d8317f587c63a571e14ffd0acc05d3171ea435 to e08f90287e9edfe06a97626196c78bbcdbeaa0a4

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

5b8e8ddSupport downloading of symlinked tarballs
5ece103Add a new gfortran spkg and use it when gfortran is not found
e08f902New configure tarball

comment:458 Changed 11 months ago by jdemeyer

  • Description modified (diff)

comment:459 Changed 11 months ago by jdemeyer

  • Description modified (diff)

comment:460 Changed 11 months ago by git

  • Commit changed from e08f90287e9edfe06a97626196c78bbcdbeaa0a4 to b4d3067b5666bfa5f2208e8405d7cec4706e3738

Branch pushed to git repo; I updated commit sha1. New commits:

b4d3067Mark gfortran as not installed after installing gcc

comment:461 Changed 11 months ago by jdemeyer

On Darwin osx 16.7.0 Darwin Kernel Version 16.7.0: Mon Nov 13 21:56:25 PST 2017; root:xnu-3789.72.11~1/RELEASE_X86_64 x86_64:

----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 6444.0 seconds
    cpu time: 33345.3 seconds
    cumulative wall time: 48795.6 seconds

So I think that this ticket is good to go.

comment:462 follow-up: Changed 11 months ago by fbissey

Thanks Jeroen, a few interesting things in there I wouldn't have thought about. So you think it can go in spite of eclib's test suite breaking? Upstream has a fix getting ready for that but I understand changes will be needed sage side to match.

comment:463 follow-up: Changed 11 months ago by dimpase

I've checked that FC=gfortran6 can be provided on FreeBSD.

One more thing for a followup ticket: sort out the case of /bin/sh not being bash (as on FreeBSD) and running with bash as the SHELL.Curently this leads to /bin/sh creeping up in configure, and a breakage.

Last edited 11 months ago by dimpase (previous) (diff)

comment:464 in reply to: ↑ 462 Changed 10 months ago by jdemeyer

Replying to fbissey:

So you think it can go in spite of eclib's test suite breaking?

That is really more a question for the release manager. Anyway, it's far more important that the Sage doctests pass.

comment:465 in reply to: ↑ 463 Changed 10 months ago by jdemeyer

Replying to dimpase:

One more thing for a followup ticket: sort out the case of /bin/sh not being bash (as on FreeBSD)

See #23448.

comment:466 Changed 10 months ago by dimpase

  • Status changed from needs_review to positive_review

I think it's ready. (I ran more tests on OSX).

comment:467 Changed 10 months ago by jdemeyer

I remind you that there is still the dependency #24476 to be reviewed.

comment:468 Changed 10 months ago by dimpase

  • Milestone changed from sage-8.1 to sage-8.2

comment:469 Changed 10 months ago by vbraun

  • Status changed from positive_review to needs_work

Merge conflict

comment:470 Changed 10 months ago by git

  • Commit changed from b4d3067b5666bfa5f2208e8405d7cec4706e3738 to d327bf6e636f3da1827d18e67bf1f2d1e6829892

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

d327bf6Mark gfortran as not installed after installing gcc

comment:471 Changed 10 months ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to positive_review

Guessing that the conflict comes from the configure tarball, I just removed that commit.

comment:472 Changed 10 months ago by vbraun

  • Status changed from positive_review to needs_work

merge conflict

comment:473 Changed 10 months ago by git

  • Commit changed from d327bf6e636f3da1827d18e67bf1f2d1e6829892 to 2c2ca7bfa18bdda3d5ae7653c5b67fa462814a09

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

c93f7f1Add a new gfortran spkg and use it when gfortran is not found
2c2ca7bMark gfortran as not installed after installing gcc

comment:474 Changed 10 months ago by jdemeyer

  • Status changed from needs_work to positive_review

Rebased to 8.2.beta3

comment:475 follow-up: Changed 10 months ago by vbraun

  • Status changed from positive_review to needs_work

Tries to build gfortran on Debian 7 (despite being installed) but fails:

[gfortran-7.2.0] Found local metadata for gfortran-7.2.0
[gfortran-7.2.0] Using cached file /home/buildbot/slave/sage_git/build/upstream/gcc-7.2.0.tar.xz
[gfortran-7.2.0] gfortran-7.2.0
[gfortran-7.2.0] ====================================================
[gfortran-7.2.0] Setting up build directory for gfortran-7.2.0
[gfortran-7.2.0] Finished extraction
[gfortran-7.2.0] No patch files found in ../patches
[gfortran-7.2.0] ****************************************************
[gfortran-7.2.0] Host system:
[gfortran-7.2.0] Linux sagebd07_64s02 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.65-3+deb9u2~bpo8+1 (2017-01-05) x86_64 GNU/Linux
[gfortran-7.2.0] ****************************************************
[gfortran-7.2.0] C compiler: gcc
[gfortran-7.2.0] C compiler version:
[gfortran-7.2.0] Using built-in specs.
[gfortran-7.2.0] COLLECT_GCC=gcc
[gfortran-7.2.0] COLLECT_LTO_WRAPPER=/home/buildbot/slave/sage_git/build/local/libexec/gcc/x86_64-pc-linux-gnu/7.2.0/lto-wrapper
[gfortran-7.2.0] Target: x86_64-pc-linux-gnu
[gfortran-7.2.0] Configured with: ../src/configure --prefix=/home/buildbot/slave/sage_git/build/local --with-local-prefix=/home/buildbot/slave/sage_git/build/local --with-gmp=/home/buildbot/slave/sage_git/build/local --with-mpfr=/home/buildbot/slave/sage_git/build/local --with-mpc=/home/buildbot/slave/sage_git/build/local --with-system-zlib --disable-multilib --disable-nls --enable-languages=c,c++,fortran --disable-libitm  
[gfortran-7.2.0] Thread model: posix
[gfortran-7.2.0] gcc version 7.2.0 (GCC) 
[gfortran-7.2.0] ****************************************************
[gfortran-7.2.0] Error: gcc is already installed
[gfortran-7.2.0] 
[gfortran-7.2.0] real	0m0.003s
[gfortran-7.2.0] user	0m0.000s
[gfortran-7.2.0] sys	0m0.000s
[gfortran-7.2.0] ************************************************************************
[gfortran-7.2.0] Error installing package gfortran-7.2.0
[gfortran-7.2.0] ************************************************************************

For the record:

buildbot@sagebd07_64s02:~/slave/sage_git/build$ gfortran --version
GNU Fortran (Debian 4.7.2-5) 4.7.2

comment:476 in reply to: ↑ 475 Changed 10 months ago by dimpase

Replying to vbraun:

It's not 100% clear what the bot configuration really is. Does it install gcc from Sage, too? If not, what is gcc installed? Does it match gfortran installed, version-vise?

(all this can be found in the output of the main ./configure script.)

comment:477 follow-ups: Changed 10 months ago by fbissey

Considering the output, I'd say it is an incremental build. sage's gcc is installed because the OS' one is too old (4.7.2). So there is something in the configure logic that did go wrong and I thought I had guarded and tested against it. At least spkg-install behaved as expected and stopped before causing further dommages.

comment:478 in reply to: ↑ 477 Changed 10 months ago by dimpase

Replying to fbissey:

Considering the output, I'd say it is an incremental build. sage's gcc is installed because the OS' one is too old (4.7.2). So there is something in the configure logic that did go wrong and I thought I had guarded and tested against it. At least spkg-install behaved as expected and stopped before causing further dommages.

perhaps just change spkg-install to exit happily if gcc is already installed?

On the other hand it seems that one simply should not do incremental build where this ticket is tested.

comment:479 in reply to: ↑ 477 Changed 10 months ago by jdemeyer

Replying to fbissey:

So there is something in the configure logic that did go wrong

I think this is again the typical issue that configure doesn't get updated on the buildbots. The bug is in the release manager workflow, not this ticket. I'll try to provide a workaround.

comment:480 Changed 10 months ago by jdemeyer

  • Dependencies changed from #24476 to #24579
  • Status changed from needs_work to positive_review

comment:481 Changed 10 months ago by git

  • Commit changed from 2c2ca7bfa18bdda3d5ae7653c5b67fa462814a09 to ac7816d0be8d72884422d107e361889ca3400fd4
  • Status changed from positive_review to needs_review

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

79c3b7aPreparations for adding a gfortran package
ac7816dAdd a new gfortran spkg and use it when gfortran is not found

comment:482 Changed 10 months ago by jdemeyer

  • Milestone changed from sage-8.2 to sage-pending
  • Status changed from needs_review to positive_review

comment:483 Changed 10 months ago by vbraun

Is installed and matches version

buildbot@sagebd07_64s02:~$ gcc --version
gcc (Debian 4.7.2-5) 4.7.2
[...]
buildbot@sagebd07_64s02:~$ gfortran --version
GNU Fortran (Debian 4.7.2-5) 4.7.2

comment:484 Changed 10 months ago by vbraun

  • The error happend with both incremental and full build
  • Only happens on Debian 7, not other distros

comment:485 Changed 10 months ago by fbissey

The bit that's supposed to prevent this from happening in configure is

# if GCC needs to be installed GFORTRAN shouldn't be installed
# GCC will provide GFORTRAN in that case
if test $need_to_install_gcc = yes ; then
    need_to_install_gfortran=no
fi

so this test must be failing with the shell used debian 7. It will to be converted into something more universally reliable.

comment:486 Changed 10 months ago by dimpase

Could it be that on that Debian 7 system /bin/sh is not bash? This might explain this singularity.

comment:487 Changed 10 months ago by jdemeyer

Please read 479 for a possible explanation and see #24579 for a possible fix...

comment:488 Changed 10 months ago by dimpase

Why is this in sage-pending?

comment:489 Changed 10 months ago by jdemeyer

  • Milestone changed from sage-pending to sage-8.2

comment:490 Changed 10 months ago by vbraun

  • Branch changed from u/jdemeyer/clang-progress to ac7816d0be8d72884422d107e361889ca3400fd4
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:491 Changed 10 months ago by fbissey

  • Commit ac7816d0be8d72884422d107e361889ca3400fd4 deleted

Thanks everyone!

Note: See TracTickets for help on using tickets.