Opened 4 years ago

Closed 2 years ago

Last modified 2 years ago

#17635 closed enhancement (fixed)

Update Givaro, FFLAS-FFPACK and LinBox

Reported by: jpflori Owned by:
Priority: major Milestone: sage-7.4
Component: packages: standard Keywords: linbox, givaro, fflas-ffpack, sd75
Cc: fbissey, cpernet Merged in:
Authors: Clément Pernet, Jeroen Demeyer Reviewers: François Bissey, Jeroen Demeyer, Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 7e8eb3a (Commits) Commit:
Dependencies: Stopgaps:

Attachments (5)

fflas_ffpack-2.2.0.log (46.6 KB) - added by jdemeyer 3 years ago.
fflas_ffpack build failure on x86_64 GNU/linux GCC 4.8.4
fflas-ffpack-simd32bits.patch (2.1 KB) - added by cpernet 3 years ago.
Patch for the 32 bits AVX bug
giva.log (3.6 KB) - added by dimpase 2 years ago.
miniaes error log (gcc 6.1.1)
linbox.log (63.2 KB) - added by chapoton 2 years ago.
shorter linbox log
linbox-1.4.2.log (39.9 KB) - added by dcoudert 2 years ago.
linbox log when trying the trick of comment https://trac.sagemath.org/ticket/17635#comment:221

Download all attachments as: .zip

Change History (260)

comment:1 Changed 4 years ago by jpflori

Requires to update:

  • fflas-ffpack to 2.0.0, see #17636
  • linbox, no release yet

comment:2 follow-up: Changed 4 years ago by fbissey

  • Cc fbissey added

So we have to wait for linbox? Do we have an idea of when it will happen?

comment:3 in reply to: ↑ 2 ; follow-up: Changed 4 years ago by jpflori

Replying to fbissey:

So we have to wait for linbox? Do we have an idea of when it will happen?

Yes we have to wait for a new Linbox, expected about the end of february, Clément Pernet wrote me, together with updated Givaro and fflas-ffpack.

By the way, I started this for #17630 which might interest you as well, I'll push code there tomorrow.

comment:4 in reply to: ↑ 3 Changed 4 years ago by jdemeyer

  • Description modified (diff)
  • Milestone changed from sage-6.5 to sage-6.7
  • Summary changed from Update Givaro to 3.8.0 to Update Givaro, FFLAS-FFPACK and LinBox

Replying to jpflori:

Replying to fbissey:

So we have to wait for linbox? Do we have an idea of when it will happen?

Yes we have to wait for a new Linbox, expected about the end of february, Clément Pernet wrote me, together with updated Givaro and fflas-ffpack.

It's June already...

Since givaro, LinBox and FFLAS-FFPACK all depend on each other, we should make one ticket.

Last edited 3 years ago by jdemeyer (previous) (diff)

comment:5 Changed 3 years ago by jdemeyer

  • Description modified (diff)
  • Milestone changed from sage-6.7 to sage-6.8

comment:6 follow-up: Changed 3 years ago by cpernet

  • Cc cpernet added

Sorry for the delay in releasing the fflas-ffpack/Givaro/LinBox environment. We're working on it now, and I'll keep you posted about the status. It should happen before the end of the year and hopefully much earlier.

I also meant to propose to replace ATLAS by OpenBLAS in sage. Looking at your clean-up work of the BLAS in #17630, I understand that we should proceed in the following order:

  1. release linbox
  2. upgrade linbox fflas-ffpack Givaro in Sage (this ticket #17635)
  3. clean-up the BLAS usage #17630
  4. replace ATLAS by OpenBLAS (ticket to be created)

I'll be happy to help in these tasks.

Last edited 3 years ago by cpernet (previous) (diff)

comment:7 Changed 3 years ago by jdemeyer

Let me remind you also about #15535, which pops up regularly. I don't know if it's a bug in Linbox or in the way that Sage uses Linbox, but it would be good to have a Linbox developer look at it.

comment:8 in reply to: ↑ 6 Changed 3 years ago by fbissey

Replying to cpernet:

Sorry for the delay in releasing the fflas-ffpack/Givaro/LinBox environment. We're working on it now, and I'll keep you posted about the status. It should happen before the end of the year and hopefully much earlier.

I also meant to propose to replace ATLAS by OpenBLAS in sage. Looking at your clean-up work of the BLAS in #17630, I understand that we should proceed in the following order:

  1. release linbox
  2. upgrade linbox fflas-ffpack Givaro in Sage (this ticket #17635)
  3. clean-up the BLAS usage #17630
  4. replace ATLAS by OpenBLAS (ticket to be created)

I'll be happy to help in these tasks.

I'll admit to have been a bit demotivated on the blas cleaning front. The tickets needs re-organizing and have good descriptions so we know what is supposed to go in them. Getting ATLAS to produce .pc file and using them is probably a priority task. Once that's done someone could create an optional openblas spkg or something.

comment:9 follow-ups: Changed 3 years ago by cpernet

  • Dependencies set to 18323

I'm working on it, and will release a new version of the 3 libs for the occasion, so as to ensure a smoother integration upstream. This will very likely be givaro-4.0.1 fflas-ffpack-2.2.0 linbox-1.4.1

These libraries now require c++11 support and this tickets therefore depends on #18323.

Inspired by what's propose in there, I suggest to add lines of the form

+# distutils: extra_compile_args = -std=c++11

to libs/linbox/linbox.pxd lib/linbox/fflas.pxd lib/linbox/echelonform.pxd but I understand that this is subject to a debate on sage-devel https://groups.google.com/forum/#!topic/sage-devel/aGwotQA9to8

If this option happens to be accepted, we would then need to add more cflags in the list: namely those produced by fflas-ffpack configure related to the target architecture: -mavx -msse4.1 and others like -fabi-version=6 -fopenmp.

I could not figure a way to append the string returned by bin/fflas-ffpack-config --cflags to these pxd files. Any hint?

Last edited 3 years ago by cpernet (previous) (diff)

comment:10 Changed 3 years ago by cpernet

  • Dependencies changed from 18323 to #18323

comment:11 in reply to: ↑ 9 Changed 3 years ago by fbissey

Replying to cpernet:

I could not figure a way to append the string returned by bin/fflas-ffpack-config --cflags to these pxd files. Any hint?

Add a definition in module_list.py, you define a LINBOX_FLAGS. It would be easier if you had a .pc file as we have a python interface to pkgconfig that we can use easily. Here you have to do your own parsing from python. Anyway you can then use LINBOX_FLAGS in the .pxd file.

comment:12 in reply to: ↑ 9 ; follow-up: Changed 3 years ago by jdemeyer

Replying to cpernet:

-mavx -msse4.1

I understand that the libraries themselves want these, but why the Cython files?

and others like -fabi-version=6

This looks like it should be used always for all C++ files then. Mixing ABI's doesn't look like a good idea.

-fopenmp

Same question as before, is this really needed for the Cython interface?

comment:13 in reply to: ↑ 12 Changed 3 years ago by cpernet

Replying to jdemeyer:

Replying to cpernet:

-mavx -msse4.1

I understand that the libraries themselves want these, but why the Cython files?

fflas-ffpack and linbox are essentially source libraries, so when you include the headers, you include the whole code. This is by design, due to the ubiquity of template programming in these libraries.

Now the old linbox-sage interface compiles some specializations and offer them through a C api. These extra flags would not be required if we would stick to using this interface.

But, since then, Cython started to offer the capability to hook C++ code directly and more functionnalities have been added in sage where fflas-ffpack code is called directly, like in libs/linbox/fflas.pxd where the C++ routine FFLAS::fgemm is called.

and others like -fabi-version=6

This looks like it should be used always for all C++ files then. Mixing ABI's doesn't look like a good idea.

Sure. Maybe one should change the ABI higher up then.

-fopenmp

Same question as before, is this really needed for the Cython interface?

Same answer as above.

comment:14 Changed 3 years ago by jdemeyer

  • Dependencies changed from #18323 to #19298

comment:15 Changed 3 years ago by cpernet

  • Branch set to u/cpernet/givaro_fflasffpack_linbox

comment:16 follow-up: Changed 3 years ago by cpernet

  • Commit set to dd81c26b4ece4e206e5c3eb8c42e5a22337bc458

I added my working branch to the ticket. I could not find if and where I should store the tarballs of the new spkg?

As of today (sept 28 2015); they correspond to the upstream version of the 3 libs: givaro, fflas-ffpack and linbox.

I added .pc files for givaro and fflas-ffpack (coming soon for linbox), and defined the extra_compile_flages in modules_list.py as suggested by fbissey.

This is still work in progress:

  • LinBox sparse solver make the test-suite fail: I'm investigating the problem, which I could not reproduce in LinBox for the moment.
  • the extra_compile_flags should be inferred from the .pc files (harcoded for the moment). I did not find how to use pkg-config through python to do it.

New commits:

dd81c26Update givaro to v4.0.1 fflas-ffpack to v2.2.0 and linbox to v1.4.1rc0.

comment:17 in reply to: ↑ 16 Changed 3 years ago by jdemeyer

Replying to cpernet:

I could not find if and where I should store the tarballs of the new spkg?

Just add a link to the upstream tarballs on this ticket. If any renaming is needed, you should rename the tarball and add a link to the renamed tarball.

comment:18 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:19 Changed 3 years ago by git

  • Commit changed from dd81c26b4ece4e206e5c3eb8c42e5a22337bc458 to c97f54702ff9273a6fd17954aa3578198300124c

Branch pushed to git repo; I updated commit sha1. New commits:

c97f547update fflas-givaro-linbox spkg version and minor chnages to configuration files

comment:20 Changed 3 years ago by jdemeyer

Instead of adding flags to individual extensions, they should be added to the .pxd or .pyx files.

comment:21 follow-up: Changed 3 years ago by cpernet

Ok, now I get confused: fbissey suggested to define all compilation flags in module_list.py. I saw there that most spkg related extension had their flags set in this file: e.g. flint, fplll, Lfunction, m4ri, m4rie... Hence I moved all flags that I originally wrote in the linbox.pxd fflas-ffpack.pxd givaro.pxd files to module_list.py which has the advantage of defining everything at the same place.

Jeroen or François are you suggesting that the proper way to proceed is to

  • define the flags givaro_extra_compile_args, fflas_ffpack_extra_compile_args, linbox_extra_compile_args in module_list.py
  • use them in the pxd files with something like
    +# distutils: extra_compile_args = ${givaro_extra_compile_args}
    

? Will do, if that's the way to go.

Also, I did not find how to invoque the python interface to pkgconfig, mentionned by françois.

comment:22 Changed 3 years ago by fbissey

Sorry I have been of the net for most of the week. For how to use pkgconfig you can follow the way I use it for blas in sage-on-gentoo at https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/sage/files/sage-6.8-blas-r1.patch. Note that you probably don't need to parse the library_dir, I do it for blas in unusual location, which is expected for intel MKL or the AMD equivalent.

comment:23 in reply to: ↑ 21 Changed 3 years ago by jdemeyer

Replying to cpernet:

I moved all flags that I originally wrote in the linbox.pxd fflas-ffpack.pxd givaro.pxd files to module_list.py

Please don't!

which has the advantage of defining everything at the same place.

...but the disadvantage of requiring to add the flags to every single extension, which is much less scalable.

Jeroen or François are you suggesting that the proper way to proceed is to

  • define the flags givaro_extra_compile_args, fflas_ffpack_extra_compile_args, linbox_extra_compile_args in module_list.py
  • use them in the pxd files with something like
    +# distutils: extra_compile_args = ${givaro_extra_compile_args}
    

Essentially yes. You can grep for GSL_LIBRARIES to see how it's supposed to be done.

comment:24 Changed 3 years ago by git

  • Commit changed from c97f54702ff9273a6fd17954aa3578198300124c to fb16751ee2a275d55bef8824ee606fdd98f817f8

Branch pushed to git repo; I updated commit sha1. New commits:

fb16751Pass the library specific cflags in the pxd files instead of extensions in module_list.py

comment:25 follow-up: Changed 3 years ago by jdemeyer

This is still not quite what I meant.

For example, why does src/sage/rings/polynomial/multi_polynomial_ideal_libsingular.pxd need GIVARO_CFLAGS? Looking at that file, there is nothing related to givaro.

I assume you added GIVARO_CFLAGS because it still somehow indirectly includes Givaro code. But then you should add GIVARO_CFLAGS in the place where the Givaro headers are actually included and Cython's dependency tracking should pick it up.

See again the GSL_LIBRARIES example: I don't add GSL_LIBRARIES to every module which somehow indirectly uses GSL, only where the library functions are actually declared.

comment:26 Changed 3 years ago by git

  • Commit changed from fb16751ee2a275d55bef8824ee606fdd98f817f8 to c25de3330e09067dff47ecd8330df5ba7a00746e

Branch pushed to git repo; I updated commit sha1. New commits:

c25de33Remove all unecessary extra_compile_args for GIVARO and FFLASFFPACK.

comment:27 Changed 3 years ago by git

  • Commit changed from c25de3330e09067dff47ecd8330df5ba7a00746e to 73cc6b328406860fad586dea893625bac3f7f1fa

Branch pushed to git repo; I updated commit sha1. New commits:

73cc6b3Finds GIVARO and FFLASFFPACK CFLAGS in the .pc pkgconfig file.

comment:28 in reply to: ↑ 25 Changed 3 years ago by cpernet

Replying to jdemeyer:

I assume you added GIVARO_CFLAGS because it still somehow indirectly includes Givaro code. But then you should add GIVARO_CFLAGS in the place where the Givaro headers are actually included and Cython's dependency tracking should pick it up.

Yes. It was necessary to add these flags to extensions when everything was gathered in module_list.py, but no longer now that it is in each pxd files. I now see the point! Sorry for messing this up, it's fixed. I also made the CFLAGS variables set by the pkgconfig .pc files, as suggested by François.

One pb remains: there is no gmp.pc for the moment, and the givaro.pc lists gmp as a dependency. Similarly fflas-ffpack.pc should list blas as dependency, but we still don't have a blas.pc file (see #17075). I suggest to remove these dependencies for the moment, and I'll add them once we have the proper .pc files. If you confirm, I'll produce new tar.gz for givaro and fflas-ffpack accordingly.

comment:29 Changed 3 years ago by git

  • Commit changed from 73cc6b328406860fad586dea893625bac3f7f1fa to 7fd9523fad7dd778ee1d42c803b06f103ccdd709

Branch pushed to git repo; I updated commit sha1. New commits:

7fd9523Add link flags for fflas-ffpack.

comment:30 Changed 3 years ago by jdemeyer

Are there any significant API changes in the new linbox? I would like to add a linbox interface for sparse matrices over ZZ. I could do that using the current linbox-1.3.2 but I'd like to know that it will still work after upgrading linbox.

comment:31 Changed 3 years ago by jdemeyer

  • Branch changed from u/cpernet/givaro_fflasffpack_linbox to u/jdemeyer/givaro_fflasffpack_linbox

comment:32 Changed 3 years ago by jdemeyer

  • Commit changed from 7fd9523fad7dd778ee1d42c803b06f103ccdd709 to c4473522fac04f2726d7406b9fbf2fdcd0796f8b
  • Dependencies #19298 deleted
  • Milestone changed from sage-6.8 to sage-7.1

Merged latest beta.


New commits:

c447352Merge tag '7.1.beta3' into t/17635/givaro_fflasffpack_linbox

comment:33 Changed 3 years ago by git

  • Commit changed from c4473522fac04f2726d7406b9fbf2fdcd0796f8b to f31752c6865b53eeef48d4fc7a46280823a7494e

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

f31752cMerge tag '7.1.beta3' into t/17635/givaro_fflasffpack_linbox

comment:34 Changed 3 years ago by jdemeyer

fflas_ffpack doesn't build:

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I../.. -I../../fflas-ffpack/utils/ -I../../fflas-ffpack/fflas/ -I../../fflas-ffpack/ffpack -I../../fflas-ffpack/field -I/usr/local/src/sage-git/local/include -I/usr/local/src/sage-git/local//include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mavx -fabi-version=6 -fopenmp -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -g -O2 -fPIC -std=gnu++11 -c fflas_L1_inst.C  -fPIC -DPIC -o .libs/fflas_L1_inst.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I../.. -I../../fflas-ffpack/utils/ -I../../fflas-ffpack/fflas/ -I../../fflas-ffpack/ffpack -I../../fflas-ffpack/field -I/usr/local/src/sage-git/local/include -I/usr/local/src/sage-git/local//include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mavx -fabi-version=6 -fopenmp -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -g -O2 -fPIC -std=gnu++11 -c fflas_L3_inst.C  -fPIC -DPIC -o .libs/fflas_L3_inst.o
In file included from ../../fflas-ffpack/fflas/fflas_sparse/csr.h:89:0,
                 from ../../fflas-ffpack/fflas/fflas_sparse.h:126,
                 from ../../fflas-ffpack/fflas/fflas.h:164,
                 from fflas_L1_inst.C:34:
../../fflas-ffpack/fflas/fflas_sparse/csr/csr_pspmm.inl:48:2: error: stray '#' in program
   PAR_BLOCK{
  ^

comment:35 Changed 3 years ago by cpernet

  • Branch changed from u/jdemeyer/givaro_fflasffpack_linbox to u/cpernet/givaro_fflasffpack_linbox

comment:36 Changed 3 years ago by cpernet

  • Commit changed from f31752c6865b53eeef48d4fc7a46280823a7494e to 4cd2289c8711642d448883ee6d843524ed5da5d5
  • Description modified (diff)

I've updated the tarballs of the 3 libraries to the latests rc1 (the failing code in longer there) and updated the checksums. Everything builds on my box.


New commits:

db82895Patch merged upstream in fflas-ffpack
a45e379missing coma
4cd2289updating givaro, fflas-ffpack and linbox to latest rc1 version

comment:37 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:38 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:39 Changed 3 years ago by jdemeyer

On powerpc64le, I had to kill the Givaro test test-rmint_inv_ after 11 hours. I don't think the test is supposed to take that long...

comment:40 Changed 3 years ago by cpernet

Thanks for raising this one. Bug reported on givaro's issue tracker: https://github.com/linbox-team/givaro/issues/6

Currently looking for an acces to a ppc64el to debug it.

The part of givaro that fails, recint, does small lenght multi-precision arithmetic, and very likely triggers endianness issues. Note that this part of the library is not currently used in sage neither directly nor through fflas-ffpack and linbox. However we'd like to have it fixed anyway, in order to update givaro's debian package.

Last edited 3 years ago by cpernet (previous) (diff)

comment:41 follow-up: Changed 3 years ago by jdemeyer

On the same machine, fflas-ffpack also fails: https://github.com/linbox-team/fflas-ffpack/issues/16

comment:42 Changed 3 years ago by jdemeyer

Apart from the build failures on some systems, are there any further obstructions to this ticket?

comment:43 in reply to: ↑ 41 ; follow-ups: Changed 3 years ago by cpernet

Replying to jdemeyer:

On the same machine, fflas-ffpack also fails: https://github.com/linbox-team/fflas-ffpack/issues/16

Fixed in https://github.com/linbox-team/fflas-ffpack/commit/8634b80bd0c20c538faf9471c8e2289c5a50a925

Replying to jdemeyer:

Apart from the build failures on some systems, are there any further obstructions to this ticket?

Apart from these, no obstruction on our side.

comment:44 in reply to: ↑ 43 Changed 3 years ago by jdemeyer

Replying to cpernet:

Apart from these, no obstruction on our side.

So why was this ticket never set to "needs_review" then?

comment:45 in reply to: ↑ 43 Changed 3 years ago by jdemeyer

Replying to cpernet:

Replying to jdemeyer:

On the same machine, fflas-ffpack also fails: https://github.com/linbox-team/fflas-ffpack/issues/16

Fixed in https://github.com/linbox-team/fflas-ffpack/commit/8634b80bd0c20c538faf9471c8e2289c5a50a925

Do you plan to release fflas_ffpack-2.2.0rc2 for this? Or should we just use git master?

comment:46 follow-up: Changed 3 years ago by cpernet

  • the bug of Givaro on powerpc64le has been fixed upstream : https://github.com/linbox-team/givaro/issues/6
  • yes I will cut a rc2 for both givaro and fflas-ffpack, if not a release.
  • I did not put the need_review as there was still a hanging question about the blas.pc file and gmp.pc. I assume we will do without these files for the moment.

comment:47 Changed 3 years ago by jdemeyer

  • Authors set to Clement Pernet

comment:48 in reply to: ↑ 46 Changed 3 years ago by jdemeyer

Replying to cpernet:

  • I did not put the need_review as there was still a hanging question about the blas.pc file and gmp.pc.

I see. However, did anything change compared to earlier versions of these packages regarding compilation? If not and it worked before, it should still work...

comment:49 Changed 3 years ago by cpernet

As far as build and compilation is concerned, nothing changed. I'm checking a last few things on the linbox side (test suite fails badly on power8). Then I'd like to release this evening or tomorow and put this ticket in needs_review status.

comment:50 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:51 Changed 3 years ago by git

  • Commit changed from 4cd2289c8711642d448883ee6d843524ed5da5d5 to f82a77dc434c9e0c93aac406f239285cc140295a

Branch pushed to git repo; I updated commit sha1. New commits:

f82a77dUpdate checksums to the latests released packages

comment:52 Changed 3 years ago by cpernet

  • Status changed from new to needs_review

Givaro, fflas-ffpack and linbox have just been released: https://groups.google.com/forum/#!topic/linbox-devel/6dIj_HW3k38

I updated the link in the ticket descriptions to the latest tarball and updated the checksums.

This ticket is now ready for review.

comment:53 Changed 3 years ago by fbissey

Before passing to other details for this ticket, is everyone ok with linbox having openmp enabled?

comment:54 Changed 3 years ago by fbissey

Actually both fflas-ffpack and linbox, fflas-ffpack has openmp enabled by default and it is stated explicitly in linbox. Our minmal compiler should have openmp support, I am wondering about unintended consequences. In sage-on-gentoo I am now using openblas with threads rather than openmp as I got into trouble with it.

comment:55 follow-up: Changed 3 years ago by cpernet

Eventhough openmp is enabled in fflas-ffpack̀ and linbox, the current code called by sage does not use any OpenMP parallelism.

If you think it is necessary, I could pass the --disable-open argument to the configure scripts. Thanks for pointing out that linbox still forces the -fopenmp evenif --disable-omp was passed to configure. We'll fix it.

Regarding the use of a threaded BLAS: it is possible to run the sequential code of fflas-ffpack on top of a parallel BLAS, but one will achieve much better performances by running the parallel code of fflas-ffpack (with OpenMP enabled) on top of a sequential BLAS. This will need further discussions when we'll want to have sage run its linear algebra kernels in parallel.

comment:56 in reply to: ↑ 55 Changed 3 years ago by fbissey

Replying to cpernet:

Eventhough openmp is enabled in fflas-ffpack̀ and linbox, the current code called by sage does not use any OpenMP parallelism.

Yes

If you think it is necessary, I could pass the --disable-open argument to the configure scripts.

At this stage I think that's an unknown that will only be determined by experience.

Thanks for pointing out that linbox still forces the -fopenmp evenif --disable-omp was passed to configure. We'll fix it.

I haven't looked at linbox' configure.ac yet so I don't know about that. I only noticed that it for fflas-ffpack it is the default if you don't pass anything.

Regarding the use of a threaded BLAS: it is possible to run the sequential code of fflas-ffpack on top of a parallel BLAS, but one will achieve much better performances by running the parallel code of fflas-ffpack (with OpenMP enabled) on top of a sequential BLAS. This will need further discussions when we'll want to have sage run its linear algebra kernels in parallel.

Sure.

Last edited 3 years ago by fbissey (previous) (diff)

comment:57 Changed 3 years ago by fbissey

I had a concern that there may be linking issue on cygwin because of libgomp. But -fopenmp should be in the CFLAGS provided by fflas-ffpack.pc. So long as those flags are passed to the linker I guess we should be ok.

@jpflori can you comment on the linking needs on cygwin?

comment:58 follow-up: Changed 3 years ago by fbissey

OK, more thoughts about fflas-ffpack. By default usage of sse4.1, avx, avx2 and fma instructions are enabled if found. This is a problem when building a binary for redistribution as we cannot guarantee that the target machine will support any instructions found on the building machine. We need to be able to disable those for binary build (SAGE_FAT_BINARY=yes?).

Going through linbox now. Like fflas-ffpack, openmp is enabled by default even without --enable-openmp in spkg-install but it looks like it can be disable the standard way to me. linbox looks for

LB_CHECK_IML
LB_CHECK_M4RI
LB_CHECK_M4RIE
LB_CHECK_MPFR
# we nedd mpfr next :
LB_CHECK_FPLLL
LB_CHECK_FLINT

LB_CHECK_NTL

but none of these are in linbox's dependencies. Are any of those actually used? Actually the current version should depend on ntl and iml

(sage-sh) fbissey@QCD-nzi3:sage-7.0$ ldd -r local/lib/liblinboxsage.so.0.0.0 
	linux-vdso.so.1 (0x00007ffcaf131000)
	libgmpxx.so.8 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgmpxx.so.8 (0x00007f3f89ea8000)
	libgmp.so.16 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgmp.so.16 (0x00007f3f89c31000)
	libntl.so.19 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libntl.so.19 (0x00007f3f89810000)
	libcblas.so.3 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libcblas.so.3 (0x00007f3f895ed000)
	libatlas.so.3 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libatlas.so.3 (0x00007f3f88d01000)
	liblinbox.so.0 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/liblinbox.so.0 (0x00007f3f88afa000)
	libgivaro.so.0 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgivaro.so.0 (0x00007f3f8889d000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6 (0x00007f3f88557000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f3f88253000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f3f87eb8000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1 (0x00007f3f87ca0000)
	libgf2x.so.1 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgf2x.so.1 (0x00007f3f87a8a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8786d000)
	/lib64/ld-linux-x86-64.so.2 (0x0000561bc940b000)
undefined symbol: maxMagnMP	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: findRNS	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: RNSbound	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: certSolveMP	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: certSolveRedMP	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: nonsingSolvLlhsMM	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: nonsingSolvRNSMM	(local/lib/liblinboxsage.so.0.0.0)

compared to sage-on-gentoo on the same machine

ldd -r /usr/lib64/liblinboxsage.so.0.0.0
	linux-vdso.so.1 (0x00007ffe013bb000)
	libgmpxx.so.4 => /usr/lib64/libgmpxx.so.4 (0x00007f8a84d7a000)
	libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00007f8a84b09000)
	libntl-9.6.2.so => /usr/lib64/libntl-9.6.2.so (0x00007f8a846e5000)
	libreflapack.so => /usr/lib64/libreflapack.so (0x00007f8a83f03000)
	libopenblas_threads.so.0 => /usr/lib64/libopenblas_threads.so.0 (0x00007f8a8396b000)
	liblinbox.so.0 => /usr/lib64/liblinbox.so.0 (0x00007f8a83763000)
	libgivaro.so.0 => /usr/lib64/libgivaro.so.0 (0x00007f8a83507000)
	libiml.so.0 => /usr/lib64/libiml.so.0 (0x00007f8a832e6000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6 (0x00007f8a82fd6000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f8a82cd2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f8a82937000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1 (0x00007f8a8271f000)
	libgf2x.so.1 => /usr/lib64/libgf2x.so.1 (0x00007f8a82509000)
	libgfortran.so.3 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgfortran.so.3 (0x00007f8a821e0000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a81fc4000)
	/lib64/ld-linux-x86-64.so.2 (0x00005557ec4cf000)
	libquadmath.so.0 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libquadmath.so.0 (0x00007f8a81d85000)

Where I had to add LIBS="-liml" for things to be completely resolved.

comment:59 Changed 3 years ago by jdemeyer

  • Reviewers set to François Bissey
  • Status changed from needs_review to needs_work

comment:60 in reply to: ↑ 58 ; follow-up: Changed 3 years ago by cpernet

Replying to fbissey:

OK, more thoughts about fflas-ffpack. By default usage of sse4.1, avx, avx2 and fma instructions are enabled if found. This is a problem when building a binary for redistribution as we cannot guarantee that the target machine will support any instructions found on the building machine. We need to be able to disable those for binary build (SAGE_FAT_BINARY=yes?).

Sure, we could do that. The alternative option would be to do as for the openblas deb package: have it include all kernel variants and let the switch happen dynamically depending on the variant of x86 it is run on. I am not very familiar with this technique and need to investigate how this can be built.

Going through linbox now. Like fflas-ffpack, openmp is enabled by default even without --enable-openmp in spkg-install but it looks like it can be disable the standard way to me.

Yes it can, but in linbox it will still pass a -fopenmp argument to g++ even if you passed the --disable-openmp option at configure time. You made me find it yesterday, and this is now https://github.com/linbox-team/linbox/issues/24.

linbox looks for

LB_CHECK_IML
LB_CHECK_M4RI
LB_CHECK_M4RIE
LB_CHECK_MPFR
# we nedd mpfr next :
LB_CHECK_FPLLL
LB_CHECK_FLINT

LB_CHECK_NTL

but none of these are in linbox's dependencies. Are any of those actually used?

These are optional dependencies. The long term goal is to have linbox code that uses these libs when available. For the moment, I think they are mostly used for benchmarking comparisons, except for NTL, IML and fplll used here and there when available.

Passing the --with-all= option to configure, as we do, will automatically enable all these optional dependencies if they are found. We could be more picky if this causes problems.

Actually the current version should depend on ntl and iml

(sage-sh) fbissey@QCD-nzi3:sage-7.0$ ldd -r local/lib/liblinboxsage.so.0.0.0 
	linux-vdso.so.1 (0x00007ffcaf131000)
	libgmpxx.so.8 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgmpxx.so.8 (0x00007f3f89ea8000)
	libgmp.so.16 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgmp.so.16 (0x00007f3f89c31000)
	libntl.so.19 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libntl.so.19 (0x00007f3f89810000)
	libcblas.so.3 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libcblas.so.3 (0x00007f3f895ed000)
	libatlas.so.3 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libatlas.so.3 (0x00007f3f88d01000)
	liblinbox.so.0 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/liblinbox.so.0 (0x00007f3f88afa000)
	libgivaro.so.0 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgivaro.so.0 (0x00007f3f8889d000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6 (0x00007f3f88557000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f3f88253000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f3f87eb8000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1 (0x00007f3f87ca0000)
	libgf2x.so.1 => /home/fbissey/sandbox/git-fork/sage-7.0/local/lib/libgf2x.so.1 (0x00007f3f87a8a000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3f8786d000)
	/lib64/ld-linux-x86-64.so.2 (0x0000561bc940b000)
undefined symbol: maxMagnMP	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: findRNS	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: RNSbound	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: certSolveMP	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: certSolveRedMP	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: nonsingSolvLlhsMM	(local/lib/liblinboxsage.so.0.0.0)
undefined symbol: nonsingSolvRNSMM	(local/lib/liblinboxsage.so.0.0.0)

compared to sage-on-gentoo on the same machine

ldd -r /usr/lib64/liblinboxsage.so.0.0.0
	linux-vdso.so.1 (0x00007ffe013bb000)
	libgmpxx.so.4 => /usr/lib64/libgmpxx.so.4 (0x00007f8a84d7a000)
	libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00007f8a84b09000)
	libntl-9.6.2.so => /usr/lib64/libntl-9.6.2.so (0x00007f8a846e5000)
	libreflapack.so => /usr/lib64/libreflapack.so (0x00007f8a83f03000)
	libopenblas_threads.so.0 => /usr/lib64/libopenblas_threads.so.0 (0x00007f8a8396b000)
	liblinbox.so.0 => /usr/lib64/liblinbox.so.0 (0x00007f8a83763000)
	libgivaro.so.0 => /usr/lib64/libgivaro.so.0 (0x00007f8a83507000)
	libiml.so.0 => /usr/lib64/libiml.so.0 (0x00007f8a832e6000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libstdc++.so.6 (0x00007f8a82fd6000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f8a82cd2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f8a82937000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgcc_s.so.1 (0x00007f8a8271f000)
	libgf2x.so.1 => /usr/lib64/libgf2x.so.1 (0x00007f8a82509000)
	libgfortran.so.3 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libgfortran.so.3 (0x00007f8a821e0000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a81fc4000)
	/lib64/ld-linux-x86-64.so.2 (0x00005557ec4cf000)
	libquadmath.so.0 => /usr/lib/gcc/x86_64-pc-linux-gnu/4.9.3/libquadmath.so.0 (0x00007f8a81d85000)

Where I had to add LIBS="-liml" for things to be completely resolved.

This seems correct: the sage-linbox interface only uses these 2 libs when found. Are you saying that the -liml option is not passed to gcc even when iml was detected at configure time? This would be another bug then.

comment:61 in reply to: ↑ 60 Changed 3 years ago by fbissey

Replying to cpernet:

This seems correct: the sage-linbox interface only uses these 2 libs when found. Are you saying that the -liml option is not passed to gcc even when iml was detected at configure time? This would be another bug then.

Yes, at least it is the case in 1.3.6, I haven't confirmed yet it is the case in 1.4.0, but if you weren't aware of it then it is probably still the case. I would rather have ntl and iml at least be in the dependencies as it makes the build reproducible. Otherwise the libraries linked to linboxsage will be random.

comment:62 follow-up: Changed 3 years ago by jdemeyer

On POWER8, I get an internal compiler error while compiling fflas-ffpack using GCC-4.9.3.

Clement, did you manage to compile fflas-ffpack on sardonis? Do you use the system-installed GCC 4.9.2 or did you compile within Sage?

comment:63 in reply to: ↑ 62 Changed 3 years ago by cpernet

Replying to jdemeyer:

On POWER8, I get an internal compiler error while compiling fflas-ffpack using GCC-4.9.3.

Clement, did you manage to compile fflas-ffpack on sardonis? Do you use the system-installed GCC 4.9.2 or did you compile within Sage?

Outch! I did compile with the system-installed gcc v 4.9.2 and did not hit any internal compiler error.

Last edited 3 years ago by cpernet (previous) (diff)

comment:64 Changed 3 years ago by fbissey

linbox again, LB_OPT is commented out in configure.ac therefore --enable-optimization which is still in spkg-install is no longer a recognised option and should be removed unless it is an error upstream.

comment:65 Changed 3 years ago by jdemeyer

src/sage/rings/finite_rings/element_givaro.pxd refers to givaro/givgfq.h, which no longer exists.

comment:66 Changed 3 years ago by jdemeyer

On powerpc64le with GCC 5.3.0, all 3 packages build and pass their testsuite.

comment:67 Changed 3 years ago by jdemeyer

On my laptop (Linux tamiyo 3.17.7-gentoo #1 SMP PREEMPT Wed Dec 31 20:06:39 CET 2014 x86_64 Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz GenuineIntel GNU/Linux), fflas_ffpack fails to build with GCC 4.8.4:

make[6]: Entering directory '/usr/local/src/sage-config/local/var/tmp/sage/build/fflas_ffpack-2.2.0/src/fflas-ffpack/interfaces/libs'
/bin/bash ../../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../../..  -I../../..  -I../../../fflas-ffpack/utils/ -I../../../fflas-ffpack/fflas/ -I../../../fflas-ffpack/ffpack -I../../../fflas-ffpack/field -I/usr/local/src/sage-config/local/include -I/usr/local/src/sage-config/local//include -D__FFLASFFPACK_HAVE_CBLAS  -msse4.1 -mavx -fabi-version=6 -fopenmp  -O2  -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -g -O2 -fPIC  -std=gnu++11 -c -o fflas_L1_inst.lo fflas_L1_inst.C
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../.. -I../../../fflas-ffpack/utils/ -I../../../fflas-ffpack/fflas/ -I../../../fflas-ffpack/ffpack -I../../../fflas-ffpack/field -I/usr/local/src/sage-config/local/include -I/usr/local/src/sage-config/local//include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mavx -fabi-version=6 -fopenmp -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -g -O2 -fPIC -std=gnu++11 -c fflas_L1_inst.C  -fPIC -DPIC -o .libs/fflas_L1_inst.o
In file included from ../../../fflas-ffpack/fflas/fflas_fger_mp.inl:40:0,
                 from ../../../fflas-ffpack/fflas/fflas.h:126,
                 from fflas_L1_inst.C:34:
../../../fflas-ffpack/fflas/fflas_fgemm/fgemm_classical_mp.inl:204:2: error: stray '#' in program
   WAIT;
  ^
../../../fflas-ffpack/fflas/fflas_fgemm/fgemm_classical_mp.inl: In function 'typename FFPACK::RNSInteger<T>::Element_ptr FFLAS::fgemm(const FFPACK::RNSInteger<T>&, FFLAS::FFLAS_TRANSPOSE, FFLAS::FFLAS_TRANSPOSE, size_t, size_t, size_t, typename FFPACK::RNSInteger<T>::Element, typename FFPACK::RNSInteger<T>::ConstElement_ptr, size_t, typename FFPACK::RNSInteger<T>::ConstElement_ptr, size_t, typename FFPACK::RNSInteger<T>::Element, typename FFPACK::RNSInteger<T>::Element_ptr, size_t, FFLAS::MMHelper<FFPACK::RNSInteger<T>, FFLAS::MMHelperAlgo::Classic, FFLAS::ModeCategories::DefaultTag, FFLAS::ParSeqHelper::Parallel<Cut, Param> >&)':
../../../fflas-ffpack/fflas/fflas_fgemm/fgemm_classical_mp.inl:204:8: error: expected ';' before string constant
   WAIT;
        ^
Makefile:576: recipe for target 'fflas_L1_inst.lo' failed
make[6]: *** [fflas_L1_inst.lo] Error 1

Full log attached.

Changed 3 years ago by jdemeyer

fflas_ffpack build failure on x86_64 GNU/linux GCC 4.8.4

comment:68 Changed 3 years ago by jdemeyer

Some of the preprocessed code looks like

#pragma omp task shared(F)
 {fadd(F, rowsize, N, A+iter.begin()*lda, lda, B+iter.begin()*ldb, ldb, C+iter.begin()*ldc, ldc);;};;} } };};} #pragma omp taskwait
;;        }

so I guess you're missing some newlines in certain macros.

comment:69 Changed 3 years ago by jdemeyer

  • Branch changed from u/cpernet/givaro_fflasffpack_linbox to u/jdemeyer/givaro_fflasffpack_linbox

comment:70 Changed 3 years ago by jdemeyer

  • Commit changed from f82a77dc434c9e0c93aac406f239285cc140295a to 6c253a017cc54fa7c6de861d6f093f5ffc9b875c

Due to https://github.com/linbox-team/linbox/issues/24 this does not actually work...


New commits:

6c253a0Disable openmp in fflas_ffpack

comment:71 Changed 3 years ago by git

  • Commit changed from 6c253a017cc54fa7c6de861d6f093f5ffc9b875c to 5dc9f122c6646c3edef0775f4c9896e42ca0e28a

Branch pushed to git repo; I updated commit sha1. New commits:

5dc9f12Disable openmp in linbox

comment:72 follow-up: Changed 3 years ago by jdemeyer

Note that this ticket conflicts badly with #20133.

comment:73 Changed 3 years ago by git

  • Commit changed from 5dc9f122c6646c3edef0775f4c9896e42ca0e28a to 35c51905b5d78fee8895b63b12d59df829096473

Branch pushed to git repo; I updated commit sha1. New commits:

26fc91bCosmetic changes
35c5190Fix Givaro includes

comment:74 Changed 3 years ago by git

  • Commit changed from 35c51905b5d78fee8895b63b12d59df829096473 to b153b931cf977d038501d9643f05f15a2c6d3f03

Branch pushed to git repo; I updated commit sha1. New commits:

b153b93Put FFLASFFPACK_CFLAGS where the linbox headers are included

comment:75 Changed 3 years ago by jdemeyer

I got this to build properly now, but there are still a lot of doctest failures in finite field arithmetic :-(

Are there any significant API changes in Givaro that we should know about?

comment:76 Changed 3 years ago by git

  • Commit changed from b153b931cf977d038501d9643f05f15a2c6d3f03 to 402dd108855f27dba8f3c8c2a7d4d1f4c3385af0

Branch pushed to git repo; I updated commit sha1. New commits:

402dd10Reduce modulo characteristic before calling GFqDom::init()

comment:77 Changed 3 years ago by jdemeyer

  • Authors changed from Clement Pernet to Clement Pernet, Jeroen Demeyer
  • Status changed from needs_work to needs_review

comment:78 Changed 3 years ago by git

  • Commit changed from 402dd108855f27dba8f3c8c2a7d4d1f4c3385af0 to 186c233af73077c98bd4b373d0adc891d0221a8a

Branch pushed to git repo; I updated commit sha1. New commits:

186c233If SAGE_FAT_BINARY is set, disable all processor-specific optimizations

comment:79 in reply to: ↑ 72 Changed 3 years ago by fbissey

Replying to jdemeyer:

Note that this ticket conflicts badly with #20133.

Will have conflicts with #20140 if I add bits to the dependencies of linbox. As mentioned earlier it has automagic dependencies on iml, ntl and now fplll (to check) so you potentially get slightly different linbox depending on the build order. I am uncomfortable with that.

comment:80 Changed 3 years ago by jdemeyer

  • Dependencies set to #20140
  • Status changed from needs_review to needs_work

I agree with your last comment.

comment:81 Changed 3 years ago by fbissey

I spent quite some today working on fflas-ffpack-2.2.0 for sage-on-gentoo, expect a PR shortly to fix the tests when fflas-ffpack has been configured with --enable-precompilation (not that the test will work if you previously installed fflas-ffpack with --enable-precompilation in a standard location - do I need to say more).

lapack, and even probably ATLAS' clapack interface, is detected properly but you have to pass all the libraries for lapack and cblas in the --with-blas-libs switch. With our new .pc files that should translate to

--with-blas-cflags="$(pkg-config cblas --cflags)" --with-blas-libs="$(pkg-config lapack --libs)"

Although it would probably be safer to put

--with-blas-cflags="$(pkg-config cblas --cflags)" --with-blas-libs="$(pkg-config lapack  cblas blas --libs)"

in case lapack.pc doesn't pull either or both cblas and blas.

sage run spkg-check after install (a bit daft but hard to fix with the current system) so we should also be ok to add a spkg-check to fflas-ffpack without including my upcoming PR.

comment:82 Changed 3 years ago by fbissey

Started on linbox. iml properly linked in this version if found. fplll detected but nowhere to be seen in the final library and no missing fplll symbols. On the other I have just been wacked by the detection of opencl that I wasn't expecting. No -lOpenCL are passed to liblinbox or liblinboxsage` when linking when it should be. Let's disable it explicitly for the time being.

comment:83 Changed 3 years ago by fbissey

I have sent a request for some explanations to Clément for the linbox tarball, I have patched something and run autoreconf on it and the libraries I get out are completely different.

comment:84 Changed 3 years ago by jdemeyer

  • Dependencies #20140 deleted
  • Milestone changed from sage-7.1 to sage-7.2

comment:85 Changed 3 years ago by jdemeyer

Any news here? There have been various changes in how Sage uses pkgconfig, should we try again to get this working?

comment:86 Changed 3 years ago by fbissey

Clement told me a couple of weeks ago he was on holidays. I also send a couple of PR for fflas-ffpack and linbox. He is supposed to cut new releases. I would prefer to wait for a new cut of linbox at least as patching the one in the description and running autoreconf on it lead to unexpected results (liblinbox and liblinboxsage were linked completely differently, WTF?).

comment:87 Changed 3 years ago by jdemeyer

  • Dependencies set to #20237

comment:88 Changed 3 years ago by git

  • Commit changed from 186c233af73077c98bd4b373d0adc891d0221a8a to 02afed245727c89ecb7437534b7555d9e1b85d6c

Branch pushed to git repo; I updated commit sha1. New commits:

0e5366fDrop -std=c99 from C++ extensions
02afed2Merge remote-tracking branch 'trac/u/jdemeyer/drop__std_c99_from_c___extensions' into t/17635/givaro_fflasffpack_linbox

comment:89 Changed 3 years ago by jdemeyer

Note that this merge is really more than just a merge: I also started using pkgconfig similarly to the way how we started using it in Sage 7.1.

comment:90 Changed 3 years ago by jdemeyer

This branch builds fine and passes make ptestlong for me.

comment:91 follow-up: Changed 3 years ago by cpernet

Hi, Sorry for the delay. Just for the status report. I've produced fflas-ffpack-2.2.1rc0 and linbox-1.4.1rc0 (links available in the description of this ticket), where I tried to fix the remaining issues:

  • fixing the disable-omp problem (issue https://github.com/linbox-team/linbox/issues/24)
  • the missing line break in OMP macro (but it should not be a problem here, as we disable omp) (comment 67_68)
  • removing the obsolete --enable-optimization of linbox (comment 64)

I still did not touch the opencl issue.

comment:92 Changed 3 years ago by cpernet

  • Branch changed from u/jdemeyer/givaro_fflasffpack_linbox to u/cpernet/givaro_fflasffpack_linbox

comment:93 in reply to: ↑ 91 Changed 3 years ago by jdemeyer

  • Commit changed from 02afed245727c89ecb7437534b7555d9e1b85d6c to ce91fb5c4af3d46c7d8cdeddfcfc7c2a8b31e69f

Replying to cpernet:

links available in the description of this ticket

No, they are not.


New commits:

608717fremove obsolete --enable-optimization
b6b60beMerge branch 'u/jdemeyer/givaro_fflasffpack_linbox' of git://trac.sagemath.org/sage into t/17635/givaro_fflasffpack_linbox
3389f1fremove obsolete --disable-optimization
ce91fb5Update to fflas-ffpack-2.2.1rc0 and linbox-1.4.1rc0.

comment:94 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:95 Changed 3 years ago by git

  • Commit changed from ce91fb5c4af3d46c7d8cdeddfcfc7c2a8b31e69f to 7823566105b8deba838cddf2ead7f174037caa39

Branch pushed to git repo; I updated commit sha1. New commits:

94d4922update to the checksum of the released givaro-4.0.1 tarball
fcbcde4update to linbox-1.4.1rc1 and fflas-ffpack-2.2.1rc1.
7823566update checksums

comment:96 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:97 Changed 3 years ago by cpernet

I updated linbox and fflas-ffpack release candidates to address the several issues on dynamic lib linkage (including francois PR). I took this opportunity to create a linbox.pc file and update the module_list.py accordingly.

all libraries compile and pass their test suites on my side, and sage builds and passes the test suite with these new rc1's. I also tested the libs but not sage compilation on sardonis (power8).

comment:98 Changed 3 years ago by fbissey

OK, so I have done my inspection:

1) list of linbox dependency that we want to be there: ntl, fplll, flint, mpfr, iml. Those are direct build dependencies

readelf -d liblinbox.so.0.0.0

Dynamic section at offset 0x44a80 contains 42 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libfflas.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libffpack.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libreflapack.so.3]
 0x0000000000000001 (NEEDED)             Shared library: [libopenblas_threads.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgivaro.so.8]
 0x0000000000000001 (NEEDED)             Shared library: [libgmpxx.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libntl-9.7.0.so]
 0x0000000000000001 (NEEDED)             Shared library: [libfplll.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libflint.so.13]
 0x0000000000000001 (NEEDED)             Shared library: [libmpfr.so.4]
 0x0000000000000001 (NEEDED)             Shared library: [libiml.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libOpenCL.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x000000000000000e (SONAME)             Library soname: [liblinbox.so.0]

flint is not in the current list. m4ri and m4rie can be removed as they are looked for by configure but not used.

2) we want to have opencl (--without-ocl) to be turned off, at least for binary build. 3) because sage/libs/linbox/linbox.pyx has been given a block

# distutils: extra_compile_args = LINBOX_CFLAGS
# distutils: libraries = LINBOX_LIBRARIES
# distutils: library_dirs = LINBOX_LIBDIR

the entry in modules_list.py doesn't need libraries = linbox_libs. cblas_include_dirs can stay as its content is somehow not included in fflas-ffpack.pc (but it is in fflas-ffpack-config --cflags). Similarly fflas-ffpack.pc has no information on the blas/lapack libraries - may be it should - you would inherit the correct stuff without having to add anything. Someone check this on a vanilla install please.

4) Do we want lapack support in fflas-ffpack? That would mean changing LINBOX_BLAS="$(pkg-config --libs cblas)" to LINBOX_BLAS="$(pkg-config --libs lapack cblas)" this has an impact on the previous item as you really need to know which libraries are used (especially if your linker is dumb). Note: you may want at least LINBOX_BLAS="$(pkg-config --cflags --libs cblas)" in case the headers are in a funny location.

comment:99 Changed 3 years ago by fbissey

@ Clement for future releases of fflas-ffpack and linbox. Now that you have committed to provide .pc files, you could insist on pkg-config being a building requirement. Once you do that, macros like FF_CHECK_GIVARO and LB_CHECK_FFLAS_FFPACK can die in peace and be replaced by

PKG_PROG_PKG_CONFIG

PKG_CHECK_MODULES(...)

leveraging pkg.m4 which is installed as part of http://pkgconfig.freedesktop.org's pkg-config. I can probably send you some PR for that in time.

Last edited 3 years ago by fbissey (previous) (diff)

comment:100 Changed 3 years ago by fbissey

  • Branch changed from u/cpernet/givaro_fflasffpack_linbox to u/fbissey/givaro_fflasffpack_linbox
  • Commit changed from 7823566105b8deba838cddf2ead7f174037caa39 to 8cd62e2c23a99466c9d22034d6a0d2db267ea771

Acted upon my comment in (1) and (2). Added some stuff in fflas_ffpack's spkg-install to address my final note in section (4). I would rather have the rest of my comments be checked/discussed before they are acted upon.


New commits:

942002dMerge branch 'develop' into givaro_fflasffpack_linbox
8cd62e2Addressing issues in in my comments section (1), (2) and final note section (4)

comment:101 follow-ups: Changed 3 years ago by fbissey

Is it just me or the givaro checksum is wrong?

Last edited 3 years ago by fbissey (previous) (diff)

comment:102 follow-up: Changed 3 years ago by fbissey

Somewhat tangential to this ticket, can someone explain to me why m4rie depends on givaro? I cannot find any reference to givaro in it, but because it is marked as depending on givaro it got rebuilt when I used make.

Of course if the dependency is bogus it shouldn't.

comment:103 in reply to: ↑ 101 Changed 3 years ago by cpernet

Replying to fbissey:

Is it just me or the givaro checksum is wrong?

My bad. I'll revert it.

comment:104 in reply to: ↑ 102 Changed 3 years ago by cpernet

Replying to fbissey:

Somewhat tangential to this ticket, can someone explain to me why m4rie depends on givaro? I cannot find any reference to givaro in it, but because it is marked as depending on givaro it got rebuilt when I used make.

Of course if the dependency is bogus it shouldn't.

It seems bogus to me. There is no reason for m4rie to depend on givaro and a grep confirms there is no mention of givaro in m4rie. The dependency can be removed.

comment:105 Changed 3 years ago by cpernet

I'm fine with your patch on (1) and (2).

(4) Regarding the use of BLAS CFLAGS. The headers of a BLAS are standard both in C and Fortran. Hence fflas-ffpack has the headers for both C and Fortran BLAS (at least for the part of it that it uses). Hence, I did not feel the need to deal with include dirs (also in the .pc file). But ok, there is no harm in adding it and it'll probably prevent some future issues.

(3) Ok to remove these entries in module_list.py. Thanks for pointing out that fflas-ffpack.pc does not mention the BLAS flags. I'll fix this.

comment:106 Changed 3 years ago by fbissey

Yes the API to cblas is "standard" that doesn't mean that the headers cannot be slightly different between implementations. You can indeed choose to include your own standard headers but if you include "system headers" for cblas you need to be more careful.

comment:107 in reply to: ↑ 101 ; follow-up: Changed 3 years ago by jdemeyer

Replying to fbissey:

Is it just me or the givaro checksum is wrong?

I remember it being wrong before too.

@cpernet: are you using a different version from the officially released one or are you changing released tarballs?

comment:108 follow-up: Changed 3 years ago by jdemeyer

The givaro tarball is missing ./configure. It looks like autogen.sh was not run...

comment:109 in reply to: ↑ 108 Changed 3 years ago by cpernet

Replying to jdemeyer:

The givaro tarball is missing ./configure. It looks like autogen.sh was not run...

Indeed, I just realized that Github release tool (simply based on a tag) does not work nicely with autotools and make dist targets generating the configure file. I added the proper tar.gz generated by make dist as a binary and will fix the link on this ticket.

comment:110 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:111 Changed 3 years ago by cpernet

  • Branch changed from u/fbissey/givaro_fflasffpack_linbox to u/cpernet/givaro_fflasffpack_linbox

comment:112 Changed 3 years ago by git

  • Commit changed from 8cd62e2c23a99466c9d22034d6a0d2db267ea771 to 33e887bb2755894c8c7a7d7fda39d7ba38a540a2

Branch pushed to git repo; I updated commit sha1. New commits:

942002dMerge branch 'develop' into givaro_fflasffpack_linbox
8cd62e2Addressing issues in in my comments section (1), (2) and final note section (4)
33e887bMerge branch 'u/fbissey/givaro_fflasffpack_linbox' into t/17635/givaro_fflasffpack_linbox

comment:113 Changed 3 years ago by jdemeyer

  • Dependencies #20237 deleted

comment:114 in reply to: ↑ 107 Changed 3 years ago by cpernet

  • Dependencies set to #20237

Replying to jdemeyer:

I remember it being wrong before too.

@cpernet: are you using a different version from the officially released one or are you changing released tarballs?

No, it's just that my scripts accidentally ran a make dist on givaro, a few commits ahead of v4.0.1, and moved the tarball to sage and had the checksums updated.

comment:115 Changed 3 years ago by jdemeyer

Found local metadata for fflas_ffpack-2.2.1rc1
Invalid checksum for cached file /usr/local/src/sage-git/upstream/fflas_ffpack-2.2.1rc1.tar.bz2, deleting
Attempting to download package fflas_ffpack-2.2.1rc1.tar.bz2 from mirrors
http://www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/fflas_ffpack/fflas_ffpack-2.2.1rc1.tar.bz2
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]
ERROR [download|run:133]: [Errno 404] Not Found: '//www.mirrorservice.org/sites/www.sagemath.org/spkg/upstream/fflas_ffpack/fflas_ffpack-2.2.1rc1.tar.bz2'

comment:116 Changed 3 years ago by jdemeyer

  • Dependencies #20237 deleted

comment:117 Changed 3 years ago by cpernet

  • Description modified (diff)

comment:118 Changed 3 years ago by git

  • Commit changed from 33e887bb2755894c8c7a7d7fda39d7ba38a540a2 to c1c84b1785dc93c3711f8589f46f4067061260be

Branch pushed to git repo; I updated commit sha1. New commits:

c1c84b1update to rc2 of fflas-ffpack-2.2.1 and rc3 of linbox-1.4.1

comment:119 Changed 3 years ago by cpernet

  • Status changed from needs_work to needs_review

@jdemeyer: sorry for this mess. I updated the packages to new rc's and updated the checksums accordingly.

This update includes a fix to fbissey's remark (3) (fflas-ffpack now exposes BLAS flags). I'll wait for a next release to complete the move to pkgconfig, (i.e. detecting givaro, fflas-ffpack only from their pc files, and requiring pkgconfig as a dependency).

This ticket is ready again on my side. I switch to needs_review. Once a reviewer confirms it works, please don't positive review too soon as it depends on the rc's for the moment. I'll need to release linbox and fflas-ffpack, and update de checksums and pkg_version before being able to close it.


New commits:

c1c84b1update to rc2 of fflas-ffpack-2.2.1 and rc3 of linbox-1.4.1

comment:120 Changed 3 years ago by jdemeyer

  • Cc embray added
  • Keywords days77 added

comment:121 Changed 3 years ago by embray

  • Cc embray removed
  • Keywords days77 removed

I'm still getting the same download error as jdemeyer. Nevermind he explained it to me.

Last edited 3 years ago by embray (previous) (diff)

comment:122 Changed 3 years ago by embray

Possibly of interest for this patch--I'm currently working getting sage building on Cygwin, and currently am stuck at fflas-ffpack with the following:

fflas_ffpack-1.6.0.p2
====================================================
Setting up build directory for fflas_ffpack-1.6.0.p2
Finished set up
****************************************************
Host system:
CYGWIN_NT-6.1 PC-pret-47 2.4.1(0.293/5/3) 2016-01-24 11:26 x86_64 Cygwin
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/lto-wrapper.exe
Target: x86_64-pc-cygwin
Configured with: /cygdrive/i/szsz/tmpp/gcc/gcc-5.3.0-3.x86_64/src/gcc-5.3.0/configure --srcdir=/cygdrive/i/szsz/tmpp/gcc/gcc-5.3.0-3.x86_64/src/gcc-5.3.0 --prefix=/usr --exec-prefix=/usr --localstatedir=/var --sysconfdir=/etc --docdir=/usr/share/doc/gcc --htmldir=/usr/share/doc/gcc/html -C --build=x86_64-pc-cygwin --host=x86_64-pc-cygwin --target=x86_64-pc-cygwin --without-libiconv-prefix --without-libintl-prefix --libexecdir=/usr/lib --enable-shared --enable-shared-libgcc --enable-static --enable-version-specific-runtime-libs --enable-bootstrap --enable-__cxa_atexit --with-dwarf2 --with-tune=generic --enable-languages=ada,c,c++,fortran,lto,objc,obj-c++ --enable-graphite --enable-threads=posix --enable-libatomic --enable-libcilkrts --enable-libgomp --enable-libitm --enable-libquadmath --enable-libquadmath-support --enable-libssp --enable-libada --enable-libgcj-sublibs --disable-java-awt --disable-symvers --with-ecj-jar=/usr/share/java/ecj.jar --with-gnu-ld --with-gnu-as --with-cloog-include=/usr/include/cloog-isl --without-libiconv-prefix --without-libintl-prefix --with-system-zlib --enable-linker-build-id --with-default-libstdcxx-abi=gcc4-compatible
Thread model: posix
gcc version 5.3.0 (GCC)
****************************************************
patching file fflas-ffpack-config.in
Using system-wide Cygwin LAPACK BLAS.
*************************************************
 Using --with-blas=-lblas
*************************************************
...
... standard ./configure junk ...
...
-----------------------------------------------
Default path = /home/embray/src/sagemath/sage/local /usr /usr/local
-----------------------------------------------
checking for GMP >= 3.1.1... found
checking whether GMP is 4.0 or greater... yes
checking whether GMP was compiled with --enable-cxx... yes
checking for GIVARO >= 30700 and < 30800... found
checking "for BLAS (-lblas)"... no
checking for C interface to BLAS with -lcblas... not working
checking for C interface to BLAS with -lblas... not working
checking for other BLAS... problem
Sorry, your BLAS are not working. Disabling.

*******************************************************************************
 ERROR: BLAS not found!

 BLAS routines are required for this library to compile. Please
 make sure BLAS are installed and specify its location with the option
 --with-blas=<lib> when running configure (or --with-cblas... see configure --help).
*******************************************************************************
Error configuring fflas-ffpack

real    0m23.800s
user    0m7.708s
sys     0m11.372s
************************************************************************
Error installing package fflas_ffpack-1.6.0.p2
************************************************************************

I tested the branch in this ticket and still had this same problem. It's a little mysterious, but I haven't deeply investigated it yet.

However, if I go into the build directory for fflas_ffpack and manually run:

$ ./configure --prefix="$SAGE_LOCAL" --libdir="$SAGE_LOCAL/lib" --with-default="$SAGE_LOCAL"  --with-blas="$(pkg-config --libs cblas)" --enable-optimization --disable-static

So ISTM it would suffice to remove the special section in spkg-install for Cygwin which I guess is no longer required (why, I don't know). That might as well be done as part of this patch.

comment:123 Changed 3 years ago by jdemeyer

Builds and tests fine for me on Linux x86_64.

comment:124 Changed 3 years ago by embray

Although I take back the above somewhat--it seems that I built my own ATLAS while building Sage. What I mentioned about the spkg-install may still be relevant for using a system BLAS library but I neglected to set SAGE_ATLAS_LIB. So I'll play around with that a bit more and open a separate issue if I find out more.

comment:125 Changed 3 years ago by fbissey

You could tweak your .pc files to point to the cygwin install. We do a particular case in the ATLAS spkg for darwin. We could consider doing one for cygwin. Especially if building sage's own ATLAS doesn't work on cygwin.

comment:126 Changed 3 years ago by embray

Building sage's ATLAS *does* work on Cygwin as long as I disable a couple bits that try to *force* me to use Cygwin's BLAS even if I didn't ask it to (and which doesn't work :) I see what you mean about Darwin though--I could try to adapt the same idea to Cygwin (but really it should only be doing this if SAGE_ATLAS_LIB is set, no?

comment:127 Changed 3 years ago by embray

On an unrelated note, when I build the linbox from this ticket on Cygwin I get:

/bin/sh ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../..  -DDISABLE_COMMENTATOR -I../../linbox -I/home/embray/src/sagemath/sage/local/include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mfma -mavx2 -fabi-version=6 -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local//include -DFFLAS_COMPILED -DFFPACK_COMPILED -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -DDISABLE_COMMENTATOR  -std=gnu++11  -g -fPIC -c -o linbox-sage.lo linbox-sage.C
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -DDISABLE_COMMENTATOR -I../../linbox -I/home/embray/src/sagemath/sage/local/include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mfma -mavx2 -fabi-version=6 -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local//include -DFFLAS_COMPILED -DFFPACK_COMPILED -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include -DDISABLE_COMMENTATOR -std=gnu++11 -g -fPIC -c linbox-sage.C  -DDLL_EXPORT -DPIC -o .libs/linbox-sage.o
linbox-sage.C:1:0: warning: -fPIC ignored for target (all code is position independent)
 /* linbox-sage.C
 ^
In file included from ../../linbox/linbox-config.h:30:0,
                 from linbox-sage.C:32:
/home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas-ffpack-config.h:42:0: warning: "_GLIBCXX_USE_C99" redefined
 #  define _GLIBCXX_USE_C99 true
 ^
In file included from /usr/lib/gcc/x86_64-pc-cygwin/5.3.0/include/c++/iostream:38:0,
                 from linbox-sage.C:27:
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/include/c++/x86_64-pc-cygwin/bits/c++config.h:1309:0: note: this is the location of the previous definition
 #define _GLIBCXX_USE_C99 1
 ^
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/as: .libs/linbox-sage.o: too many sections (46123)
/tmp/cck8OXW1.s: Assembler messages:
/tmp/cck8OXW1.s: Fatal error: can't write .libs/linbox-sage.o: File too big
/usr/lib/gcc/x86_64-pc-cygwin/5.3.0/../../../../x86_64-pc-cygwin/bin/as: .libs/linbox-sage.o: too many sections (46123)
/tmp/cck8OXW1.s: Fatal error: can't close .libs/linbox-sage.o: File too big
Makefile:503: recipe for target 'linbox-sage.lo' failed

I was able to fix this by adding CXXFLAGS=-Wa,-mbig-obj

comment:128 Changed 3 years ago by jdemeyer

Clement, I talked with François personally and we agree that the ticket is ready. So if you can make the final release of those packages, please go ahead.

I know it doesn't work on Cygwin properly, but that's true for the old version too, so it's not a regression.

comment:129 Changed 3 years ago by cpernet

  • Description modified (diff)

Ok, I have released the libraries and updated the link in the ticket description to point to the latest tarball.

comment:130 Changed 3 years ago by git

  • Commit changed from c1c84b1785dc93c3711f8589f46f4067061260be to ff19f3fec4cd238c9678a974695aefba14d6c74d

Branch pushed to git repo; I updated commit sha1. New commits:

ff19f3fupdate to the just released fflas-ffpack-2.2.1 and linbox-1.4.1.

comment:131 Changed 3 years ago by jdemeyer

  • Branch changed from u/cpernet/givaro_fflasffpack_linbox to u/jdemeyer/givaro_fflasffpack_linbox

comment:132 Changed 3 years ago by jdemeyer

  • Commit changed from ff19f3fec4cd238c9678a974695aefba14d6c74d to d0d3671efb3d418d29c01bb2def611f52d6218f8

Fixed a minor conflict in module_list.py


New commits:

d0d3671Merge tag '7.2.beta5' into t/17635/givaro_fflasffpack_linbox

comment:133 follow-up: Changed 3 years ago by fbissey

  • Reviewers changed from François Bissey to François Bissey, Jeroen Demeyer
  • Status changed from needs_review to positive_review

Hum... fell into a black hole. It has been a tough couple of weeks, let's put this in positive review and go ahead.

comment:134 in reply to: ↑ 133 Changed 3 years ago by cpernet

Replying to fbissey:

Hum... fell into a black hole. It has been a tough couple of weeks, let's put this in positive review and go ahead.

Thanks, I was just about to ping you guys about it, as I was waiting for someone else to positive review the ticket and did not see much action since my last push.

comment:136 Changed 3 years ago by fbissey

OS X: lots of problems with gmp c++ headers in givaro's test suite. givaro builds and installs fine just the tests fail to build.

g++ -DHAVE_CONFIG_H -I. -I..  -O0 -Wall -g -DNDEBUG -UGIVARO_DEBUG -UDEBUG -DSTD_RECINT_SIZE=8 -I/Users/buildslave-sage/slave/sage_git/build/local//include -I.. -I../src/kernel/system -I../src/kernel/memory -I../src/kernel/field -I../src/kernel/ring -I../src/kernel/integer -I../src/kernel -I../src/library/poly1 -I../src/kernel/bstruct -I../src/library/tools -I../src/kernel/recint  -O0 -Wall -g -DNDEBUG -UGIVARO_DEBUG -UDEBUG -DNOT_IN_PLACE -DRI_OP=addmul -DGMP_OP=mpz_addmul -DGMP_OPUI=mpz_addmul_ui -I/Users/buildslave-sage/slave/sage_git/build/local//include  -fPIC -I"/Users/buildslave-sage/slave/sage_git/build/local/include" -std=gnu++11 -c -o test_ruint_arith_addmul-test-ruint_arith.o `test -f 'test-ruint_arith.C' || echo './'`test-ruint_arith.C
In file included from ../src/kernel/recint/rudisplay.h:50:0,
                 from ../src/kernel/recint/ruint.h:73,
                 from ../src/kernel/recint/recint.h:45,
                 from test-rmint_arith.C:16:
../src/kernel/recint/ruconvert.h: In instantiation of 'mpz_class& RecInt::ruint_to_mpz(mpz_class&, const RecInt::ruint<K>&) [with long unsigned int K = 8ul; mpz_class = __gmp_expr<__mpz_struct [1], __mpz_struct [1]>]':
test-rmint_arith.C:39:29:   required from here
../src/kernel/recint/ruconvert.h:99:15: error: ambiguous overload for 'operator^=' (operand types are 'mpz_class {aka __gmp_expr<__mpz_struct [1], __mpz_struct [1]>}' and 'RecInt::limb {aka long long unsigned int}')
             a ^= static_cast<uint64_t>(*it);
               ^
../src/kernel/recint/ruconvert.h:99:15: note: candidates are:
In file included from test-rmint_arith.C:15:0:
/Users/buildslave-sage/slave/sage_git/build/local/include/gmpxx.h:3283:1: note: mpz_class& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator^=(signed char)
 __GMPZ_DEFINE_COMPOUND_OPERATOR(operator^=, __gmp_binary_xor)
 ^
/Users/buildslave-sage/slave/sage_git/build/local/include/gmpxx.h:3283:1: note: mpz_class& __gmp_expr<__mpz_struct [1], __mpz_struct [1]>::operator^=(unsigned char)
 __GMPZ_DEFINE_COMPOUND_OPERATOR(operator^=, __gmp_binary_xor)
 ^

and it goes on.

linux 32bits, the cpu is modern and offers sse4.1, avx and avx2 but is 32 bits.

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../.. -I../../../fflas-ffpack/utils/ -I../../../fflas-ffpack/fflas/ -I../../../fflas-ffpack/ffpack -I../../../fflas-ffpack/field -I/home/buildbot/slave/sage_git/build/local/include -I/home/buildbot/slave/sage_git/build/local//include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mfma -mavx2 -fabi-version=6 -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -g -O2 -fPIC -std=gnu++11 -c fflas_L2_inst.C  -fPIC -DPIC -o .libs/fflas_L2_inst.o
In file included from ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:44:0,
                 from ../../../fflas-ffpack/fflas/fflas_simd.h:208,
                 from ../../../fflas-ffpack/fflas/fflas_freduce.h:33,
                 from ../../../fflas-ffpack/fflas/fflas.h:99,
                 from fflas_L1_inst.C:34:
../../../fflas-ffpack/fflas/fflas_simd/simd256_int64.inl: In static member function 'static Simd256_impl<true, true, true, 8>::vect_t Simd256_impl<true, true, true, 8>::mulhi(Simd256_impl<true, true, true, 8>::vect_t, Simd256_impl<true, true, true, 8>::vect_t)':
../../../fflas-ffpack/fflas/fflas_simd/simd256_int64.inl:246:37: error: 'int128_t' was not declared in this scope
         return set((int128_t(ca.t[0]) * cb.t[0]) >> 64, (int128_t(ca.t[1]) * cb.t[1]) >> 64,
                                     ^
In file included from ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:44:0,
                 from ../../../fflas-ffpack/fflas/fflas_simd.h:208,
                 from ../../../fflas-ffpack/fflas/fflas_freduce.h:33,
                 from ../../../fflas-ffpack/fflas/fflas.h:99,
                 from fflas_L2_inst.C:34:
../../../fflas-ffpack/fflas/fflas_simd/simd256_int64.inl: In static member function 'static Simd256_impl<true, true, true, 8>::vect_t Simd256_impl<true, true, true, 8>::mulhi(Simd256_impl<true, true, true, 8>::vect_t, Simd256_impl<true, true, true, 8>::vect_t)':
../../../fflas-ffpack/fflas/fflas_simd/simd256_int64.inl:246:37: error: 'int128_t' was not declared in this scope
         return set((int128_t(ca.t[0]) * cb.t[0]) >> 64, (int128_t(ca.t[1]) * cb.t[1]) >> 64,
                                     ^

There may be a wrong assumptions about type there.

Changed 3 years ago by cpernet

Patch for the 32 bits AVX bug

comment:137 Changed 3 years ago by cpernet

I attached a patch that should fix the 32 bit bug where AVX instructions are available. As the only 32bit virtual box I can access to do not offer AVX instruction, I could not replicate the bug and confirm that the patch fixes it. I'm therefore waiting for Volker's confirmation before pushing it upstream.

As for the givaro OS X issue. I'll report it and look at it once we have a OSX box set up on our CI that we are currently building up.

comment:138 follow-up: Changed 3 years ago by vbraun

Can you put the patch in the branch? We have the buildbot to test tickets, why not use it.

comment:139 in reply to: ↑ 138 Changed 3 years ago by cpernet

Replying to vbraun:

Can you put the patch in the branch?

You mean, adding it to build/pkg/fflas-ffpack/patches and having the spkg-install of apply the patch ? Just asking confirmation before touching it as this change will be very temporary.

We have the buildbot to test tickets, why not use it.

I don't know how to use the buildbot for ticket related to spkg installs (i.e. how to tell buildbot where to download the tarball).

comment:140 follow-up: Changed 3 years ago by vbraun

I can run the ticket on the buildbot, but if you want me to manually patch stuff and try it then you'll probably have to wait a long time for that.

comment:141 Changed 3 years ago by fbissey

I finally got around setting a new build environment on my OS X box. I first tried to build this branch without running make check. linbox failed to build, lots of complaints about ambiguous overload of operators

libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -DDISABLE_COMMENTATOR -I../../linbox -I/Users/fbissey/build/sage-7.2.beta5/local/include -D__FFLASFFPACK_HAVE_CBLAS -msse4.1 -mfma -mavx2 -fabi-version=6 -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local//include -DFFLAS_COMPILED -DFFPACK_COMPILED -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include -I/Users/fbissey/build/sage-7.2.beta5/local/include -DDISABLE_COMMENTATOR -std=gnu++11 -g -fPIC -c linbox-sage.C  -fno-common -DPIC -o .libs/linbox-sage.o
In file included from ../../linbox/ring/ntl.h:39:0,
                 from ../../linbox/blackbox/toeplitz.h:39,
                 from ../../linbox/solutions/trace.inl:33,
                 from ../../linbox/solutions/trace.h:68,
                 from ../../linbox/algorithms/whisart_trace.h:42,
                 from ../../linbox/solutions/rank.h:45,
                 from ../../linbox/algorithms/matrix-rank.h:36,
                 from ../../linbox/algorithms/smith-form-adaptive.inl:43,
                 from ../../linbox/algorithms/smith-form-adaptive.h:106,
                 from linbox-sage.C:46:
../../linbox/ring/ntl/ntl-zz_p.h: In function 'Target& Givaro::Caster(Target&, const Source&) [with Target = Givaro::Integer; Source = NTL::ZZ_p]':
../../linbox/ring/ntl/ntl-zz_p.h:84:26: error: ambiguous overload for 'operator<<' (operand types are 'Givaro::Integer' and 'ptrdiff_t {aka long int}')
    x += Integer( txt[i] )<<(8*i) ;
                          ^
../../linbox/ring/ntl/ntl-zz_p.h:84:26: note: candidates are:
../../linbox/ring/ntl/ntl-zz_p.h:84:26: note: operator<<(int32_t {aka int}, ptrdiff_t {aka long int}) <built-in>
../../linbox/ring/ntl/ntl-zz_p.h:84:26: note: operator<<(uint32_t {aka unsigned int}, ptrdiff_t {aka long int}) <built-in>
../../linbox/ring/ntl/ntl-zz_p.h:84:26: note: operator<<(uint64_t {aka long long unsigned int}, ptrdiff_t {aka long int}) <built-in>
../../linbox/ring/ntl/ntl-zz_p.h:84:26: note: operator<<(int64_t {aka long long int}, ptrdiff_t {aka long int}) <built-in>

and there is fair amount more after that.

comment:142 Changed 3 years ago by cpernet

  • Branch changed from u/jdemeyer/givaro_fflasffpack_linbox to u/cpernet/givaro_fflasffpack_linbox

comment:143 Changed 3 years ago by git

  • Commit changed from d0d3671efb3d418d29c01bb2def611f52d6218f8 to 5d8c1d913c8b73cbac18dbc4ad5c4d0560def951

Branch pushed to git repo; I updated commit sha1. New commits:

5d8c1d9adding a patch fixing the i386 with AVX issue

comment:144 in reply to: ↑ 140 Changed 3 years ago by cpernet

Replying to vbraun:

I can run the ticket on the buildbot, but if you want me to manually patch stuff and try it then you'll probably have to wait a long time for that.

Sure no problem, I just wanted to check what was the simplest way to proceed. I have pushed the patch in the pkg/fflas-ffpack/patches dir.

comment:146 Changed 3 years ago by git

  • Commit changed from 5d8c1d913c8b73cbac18dbc4ad5c4d0560def951 to 0d2787452bd823dd1f42996206f2bcb54dc8b9ef

Branch pushed to git repo; I updated commit sha1. New commits:

0d27874Fix int size bug in the interface between singular and givaro (on 32bit arch)

comment:147 Changed 2 years ago by chapoton

  • Authors changed from Clement Pernet, Jeroen Demeyer to Clément Pernet, Jeroen Demeyer

typo in author name

comment:148 Changed 2 years ago by jpflori

  • Status changed from needs_work to needs_info

Is this back to needs_review?

comment:149 Changed 2 years ago by git

  • Commit changed from 0d2787452bd823dd1f42996206f2bcb54dc8b9ef to 20e86b3dbebb16771b9efcf1aba3feba693bfafa

Branch pushed to git repo; I updated commit sha1. New commits:

162c717preparing upgrade to linbox-1.4.2rc0 fflas-ffpack-2.2.2rc0 and givaro 4.0.2rc0
20e86b3update to the lastest releases

comment:150 Changed 2 years ago by cpernet

  • Description modified (diff)
  • Status changed from needs_info to needs_review

comment:151 Changed 2 years ago by cpernet

The 3 libraries have been released end of july but I could not update the ticket due to some git trac config issues. This is now solved, and I switched the ticket to needs review.

comment:152 follow-up: Changed 2 years ago by cpernet

  • Status changed from needs_review to needs_work

merge fails. Fix in progress

comment:153 in reply to: ↑ 152 Changed 2 years ago by leif

Replying to cpernet:

merge fails. Fix in progress

Ah, I thought I was cc'ed to this ticket...

Other issues remaining (other than rebasing / removing the C++11/14 work-arounds we meanwhile added for GCC 6.x)?

comment:154 Changed 2 years ago by cpernet

More precisely, the C++11 issue I'm having is, as I asked #21185#comment:14, to know whether with cygwin, we should still change the --std=c++11 in --std=gnu++11, as was done in #21185. If so, I'm suggesting to have

if [ "$UNAME" = CYGWIN ]; then
     CXXFLAGS="$CXXFLAGS -std=gnu++11"
fi

in spkg-install in case we need it or add nothing. Any help from someone with access to and knowledge on this cygwin bug very welcome.

comment:155 Changed 2 years ago by embray

I can confirm, the above may be needed on Cygwin. Cygwin's headers have some bugs with a number of --std=c<xx> flags.

comment:156 Changed 2 years ago by git

  • Commit changed from 20e86b3dbebb16771b9efcf1aba3feba693bfafa to 2fb5502c88d23b2e3de1a46acec7e9e1af013bfe

Branch pushed to git repo; I updated commit sha1. New commits:

2fb5502* Merging and fixing conflicts.

comment:157 Changed 2 years ago by jpflori

FYI, on my Cygwin32 and Cygwin64 setups, the provided tarballs use -std=gnu++11 by default. That's outside of Sage (and without NTL, IML), but I don't see why it would change when Sage builds it.

comment:158 Changed 2 years ago by cpernet

  • Milestone changed from sage-7.2 to sage-7.4
  • Status changed from needs_work to needs_review

I have fixed the conflicts. The cygwin issue is also solved, as noted by jpflori. There is no other issue remaining.

comment:159 Changed 2 years ago by cpernet

  • Keywords linbox givaro fflas-ffpack sd75 added

comment:160 follow-up: Changed 2 years ago by dimpase

  • Status changed from needs_review to needs_work

There is a checksum problem:

[fflas_ffpack-2.2.2] Invalid checksum for cached file /home/dima/software/sage/upstream/fflas_ffpack-2.2.2.tar.bz2, deleting

comment:161 in reply to: ↑ 160 Changed 2 years ago by cpernet

  • Status changed from needs_work to needs_review

Replying to dimpase:

There is a checksum problem:

Sorry, I had uploaded a wrong version on github. I just fixed it by updating the tarball on github. The hashes on the ticket now match.

comment:162 Changed 2 years ago by dimpase

looks good on Linux (with gcc 4.9.3 and with gcc 6.1.1, too).

comment:163 Changed 2 years ago by dimpase

sage -t --long src/sage/crypto/block_cipher/miniaes.py gives an error with gcc 6.1.1, see attachment

Changed 2 years ago by dimpase

miniaes error log (gcc 6.1.1)

comment:164 Changed 2 years ago by dimpase

  • Status changed from needs_review to needs_work

comment:165 Changed 2 years ago by git

  • Commit changed from 2fb5502c88d23b2e3de1a46acec7e9e1af013bfe to b7233c9cb64829f3c8a64b04aa744bdbb007ed27

Branch pushed to git repo; I updated commit sha1. New commits:

b7233c9Fix coercion bug: raise a TypeError exception whenever an ArithmeticError is caught, while trying to reduce an element mod the characteristic.

comment:166 Changed 2 years ago by cpernet

  • Status changed from needs_work to needs_review

This was a weird coercion bug : ArithmeticError was thrown but never caught while trying to coerce an element of GF(7) over GF(16). Not sure exactly why this bug was not triggered before.

Above commits adds a catch for this ArithmeticError and raises a TypeError which seemed to me to be the most natural way to handle it (instead of e.g. adding a catch for ArithmeticError in parent.pyx:1317).

comment:167 Changed 2 years ago by dimpase

OK, this looks good on Linux with gcc 6.1.1 now. No errors in ptestlong any more.

I'll see how it goes on OSX.

comment:168 Changed 2 years ago by dimpase

  • Reviewers changed from François Bissey, Jeroen Demeyer to François Bissey, Jeroen Demeyer, Dima Pasechnik
  • Status changed from needs_review to positive_review

look good to me.

comment:169 follow-up: Changed 2 years ago by leif

FWIW, ptestlong passes for me on x86_64 (Haswell Xeon) with GCC 6.2, despite a bunch of also not-so-harmless looking warnings, like for macro redefinitions from apparently different files... (Haven't investigated where they really originate from, just noticed them during rebuild.)

Not sure whether passing things like -mmmx :-) (and -msse* when AVX[2] is available) on a 64-bit machine is the right thingTM.


SomeoneTM should also fix src/setup.py to not pass -lfoo -lfoo -lfoo -lfoo ... to GCC / the linker.

comment:170 Changed 2 years ago by fbissey

  • Status changed from positive_review to needs_work

I don't have a compiler that default to c++11 or gun++11 and the compilation fails for sage/libs/singular/singular.cpp and sage/libs/singular/polynomial.cpp at least

[151/448] creating /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular
x86_64-pc-linux-gnu-g++ -O1 -march=native -pipe -ggdb -fPIC -I/usr/include -I/usr/include/python2.7 -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7 -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/sage/ext -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/ext -I/usr/include/python2.7 -c /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/singular.cpp -o /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/singular.o -D__STDC_LIMIT_MACROS -DNDEBUG -fno-strict-aliasing
x86_64-pc-linux-gnu-g++ -shared -Wl,-O1 -Wl,--as-needed -O1 -march=native -pipe -ggdb -O1 -march=native -pipe -ggdb /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/braiding.o -L/usr/lib -L/usr/lib64 -lbraiding -lstdc++ -lpython2.7 -o /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/lib/sage/libs/braiding.so -lpari -Ddummy
[152/448] x86_64-pc-linux-gnu-g++ -O1 -march=native -pipe -ggdb -fPIC -I/usr/lib64/python2.7/site-packages/cysignals -I/usr/include -I/usr/include/python2.7 -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7 -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/sage/ext -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/ext -I/usr/include/python2.7 -c /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/polynomial.cpp -o /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/polynomial.o -DNDEBUG -fno-strict-aliasing
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include/g++-v5/cstdint:35:0,
                 from /usr/include/givaro/udl.h:12,
                 from /usr/include/givaro/givrandom.h:22,
                 from /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/singular.cpp:370:
/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include/g++-v5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support for the \
  ^
In file included from /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/singular.cpp:370:0:
/usr/include/givaro/givrandom.h:68:16: error: invalid suffix "_ui64" on integer constant
         return _GIVRAN_MODULO_;
                ^
/usr/include/givaro/givrandom.h:77:22: error: invalid suffix "_ui64" on integer constant
             (int64_t)_GIVRAN_MULTIPLYER_
                      ^
/usr/include/givaro/givrandom.h:79:24: error: invalid suffix "_ui64" on integer constant
             % (int64_t)_GIVRAN_MODULO_ );
                        ^
/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/singular.cpp:371:27: fatal error: givaro/givgfq.h: No such file or directory
compilation terminated.
[153/448] x86_64-pc-linux-gnu-g++ -O1 -march=native -pipe -ggdb -fPIC -I/usr/include -I/usr/include/python2.7 -I/usr/lib64/python2.7/site-packages/numpy/core/include -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7 -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/sage/ext -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized -I/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/ext -I/usr/include/python2.7 -c /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/ring.cpp -o /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/temp.linux-x86_64-2.7/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/ring.o -DNDEBUG -fno-strict-aliasing
error: command 'x86_64-pc-linux-gnu-g++' failed with exit status 1
In file included from /usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include/g++-v5/cstdint:35:0,
                 from /usr/include/givaro/udl.h:12,
                 from /usr/include/givaro/givrandom.h:22,
                 from /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/polynomial.cpp:372:
/usr/lib/gcc/x86_64-pc-linux-gnu/5.3.0/include/g++-v5/bits/c++0x_warning.h:32:2: error: #error This file requires compiler and library support for the ISO C++ 2011 standard. This support is currently experimental, and must be enabled with the -std=c++11 or -std=gnu++11 compiler options.
 #error This file requires compiler and library support for the \
  ^
In file included from /scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/polynomial.cpp:372:0:
/usr/include/givaro/givrandom.h:68:16: error: invalid suffix "_ui64" on integer constant
         return _GIVRAN_MODULO_;
                ^
/usr/include/givaro/givrandom.h:77:22: error: invalid suffix "_ui64" on integer constant
             (int64_t)_GIVRAN_MULTIPLYER_
                      ^
/usr/include/givaro/givrandom.h:79:24: error: invalid suffix "_ui64" on integer constant
             % (int64_t)_GIVRAN_MODULO_ );
                        ^
/scratch2/portage/sci-mathematics/sage-9999/work/sage-9999/src-python2_7/build/cythonized/sage/libs/singular/polynomial.cpp:373:27: fatal error: givaro/givgfq.h: No such file or directory
compilation terminated.

GIVARO_CFLAGS needs to be added to the appropriate .pxd files I think.

comment:171 follow-up: Changed 2 years ago by leif

I was about to test with Sage's GCC 4.9.3 (where it will presumably fail in the same way)...

comment:172 Changed 2 years ago by fbissey

I am making and trying a patch, stay tuned.

comment:173 Changed 2 years ago by fbissey

It looks like I may have had a branch mix. Checking from scratch.

comment:174 in reply to: ↑ 171 Changed 2 years ago by leif

Replying to leif:

I was about to test with Sage's GCC 4.9.3 (where it will presumably fail in the same way)...

Hmmm, works for me, e.g. for .../sage/libs/singular/polynomial.cpp I have:

[sagelib-7.4.beta2] [ 3/12] gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -fPIC -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local/lib/python2.7/site-packages/cysignals -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local/include -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local/include/python2.7 -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local/lib/python2.7/site-packages/numpy/core/include -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/src -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/src/sage/ext -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/src/build/cythonized -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/src/build/cythonized/sage/ext -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local/include/python2.7 -c /data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/src/build/cythonized/sage/libs/singular/polynomial.cpp -o build/temp.linux-x86_64-2.7/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/src/build/cythonized/sage/libs/singular/polynomial.o -fPIC -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mfpmath=sse -fabi-version=6 -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local/include -I/data/Sage/build/devel/sage-7.4.beta2-gcc-4.4.3/local//include -fno-strict-aliasing

comment:175 Changed 2 years ago by leif

Hard to see above, but it contains

-std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mfpmath=sse -fabi-version=6

which obviously comes from Givaro.

comment:176 Changed 2 years ago by fbissey

  • Status changed from needs_work to positive_review

Definitely had some branch mixed up. So I'll put it back to positive. Sorry for the false alarm.

comment:177 in reply to: ↑ 169 Changed 2 years ago by leif

Replying to leif:

FWIW, ptestlong passes for me on x86_64 (Haswell Xeon) with GCC 6.2, despite a bunch of also not-so-harmless looking warnings, like for macro redefinitions from apparently different files... (Haven't investigated where they really originate from, just noticed them during rebuild.)

Just for the record, all (long) tests also passed with Sage's GCC 4.9.3 on the same machine (modulo the usual flaky ones which pass when rerun).

comment:178 Changed 2 years ago by vbraun

  • Status changed from positive_review to needs_work

http://build.sagedev.org/release/builders/%20%20slow%20AIMS%20%20%28Debian%208%2032%20bit%29%20incremental/builds/692/steps/compile_1/logs/stdio

[fflas_ffpack-2.2.2] libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../../.. -I../../.. -I../../../fflas-ffpack/utils/ -I../../../fflas-ffpack/fflas/ -I../../../fflas-ffpack/ffpack -I../../../fflas-ffpack/field -I/home/buildbot/slave/sage_git/build/local//include -fPIC -I/home/buildbot/slave/sage_git/build/local/include -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mfpmath=sse -fabi-version=6 -D__FFLASFFPACK_HAVE_CBLAS -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -g -O2 -fPIC -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -fabi-version=6 -c ffpack.C  -fPIC -DPIC -o .libs/ffpack.o
[fflas_ffpack-2.2.2] In file included from ../../../fflas-ffpack/fflas/fflas_simd.h:208:0,
[fflas_ffpack-2.2.2]                  from ../../../fflas-ffpack/fflas/fflas_freduce.h:33,
[fflas_ffpack-2.2.2]                  from ../../../fflas-ffpack/fflas/fflas.h:104,
[fflas_ffpack-2.2.2]                  from fflas_lvl3.C:36:
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::sll128(Simd256i_base::vect_t)':
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: error: there are no arguments to '_mm256_bslli_epi128' that depend on a template parameter, so a declaration of '_mm256_bslli_epi128' must be available [-fpermissive]
[fflas_ffpack-2.2.2]   static INLINE CONST vect_t sll128(const vect_t a) { return _mm256_bslli_epi128(a, s); }
[fflas_ffpack-2.2.2]                                                                                      ^
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::srl128(Simd256i_base::vect_t)':
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:102:85: error: there are no arguments to '_mm256_bsrli_epi128' that depend on a template parameter, so a declaration of '_mm256_bsrli_epi128' must be available [-fpermissive]
[fflas_ffpack-2.2.2]   static INLINE CONST vect_t srl128(const vect_t a) { return _mm256_bsrli_epi128(a, s); }
[fflas_ffpack-2.2.2]                                                                                      ^
[fflas_ffpack-2.2.2] Makefile:585: recipe for target 'fflas_lvl3.lo' failed
[fflas_ffpack-2.2.2] make[7]: *** [fflas_lvl3.lo] Error 1
[fflas_ffpack-2.2.2] In file included from ../../../fflas-ffpack/fflas/fflas_simd.h:208:0,
[fflas_ffpack-2.2.2]                  from ../../../fflas-ffpack/fflas/fflas_freduce.h:33,
[fflas_ffpack-2.2.2]                  from ../../../fflas-ffpack/fflas/fflas.h:104,
[fflas_ffpack-2.2.2]                  from ffpack.C:37:
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::sll128(Simd256i_base::vect_t)':
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: error: there are no arguments to '_mm256_bslli_epi128' that depend on a template parameter, so a declaration of '_mm256_bslli_epi128' must be available [-fpermissive]
[fflas_ffpack-2.2.2]   static INLINE CONST vect_t sll128(const vect_t a) { return _mm256_bslli_epi128(a, s); }
[fflas_ffpack-2.2.2]                                                                                      ^
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::srl128(Simd256i_base::vect_t)':
[fflas_ffpack-2.2.2] ../../../fflas-ffpack/fflas/fflas_simd/simd256.inl:102:85: error: there are no arguments to '_mm256_bsrli_epi128' that depend on a template parameter, so a declaration of '_mm256_bsrli_epi128' must be available [-fpermissive]
[fflas_ffpack-2.2.2]   static INLINE CONST vect_t srl128(const vect_t a) { return _mm256_bsrli_epi128(a, s); }
[fflas_ffpack-2.2.2]                                                                                      ^
[fflas_ffpack-2.2.2] Makefile:585: recipe for target 'ffpack.lo' failed
[fflas_ffpack-2.2.2] make[7]: *** [ffpack.lo] Error 1

This is Debian 8 32-bit

comment:180 in reply to: ↑ 179 Changed 2 years ago by leif

Replying to vbraun:

Same on Debian 8 64-bit http://build.sagedev.org/release/builders/%20%20slow%20AIMS%20%20%28Debian%208%2064%20bit%29%20incremental/builds/687/steps/compile_1/logs/stdio

[fflas_ffpack-2.2.2] ****************************************************
[fflas_ffpack-2.2.2] C compiler: gcc
[fflas_ffpack-2.2.2] C compiler version:
[fflas_ffpack-2.2.2] Using built-in specs.
[fflas_ffpack-2.2.2] COLLECT_GCC=gcc
[fflas_ffpack-2.2.2] COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.9/lto-wrapper
[fflas_ffpack-2.2.2] Target: x86_64-linux-gnu
[fflas_ffpack-2.2.2] Configured with: ../src/configure -v --with-pkgversion='Debian 4.9.2-10' --with-bugurl=file:///usr/share/doc/gcc-4.9/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.9 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.9 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.9-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.9-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
[fflas_ffpack-2.2.2] Thread model: posix
[fflas_ffpack-2.2.2] gcc version 4.9.2 (Debian 4.9.2-10) 
[fflas_ffpack-2.2.2] ****************************************************

Isn't that rather a Debian compiler bug?

(I've successfully built with Sage's vanilla FSF GCC 4.9.3 as well as vanilla FSF GCC 6.2.0.)

Probably we should add -fpermissive on/for Debian...

comment:181 Changed 2 years ago by leif

In any case, someone should report that upstream Debian.

comment:182 follow-up: Changed 2 years ago by dimpase

we can just blacklist gcc 4.9.2 and build our 4.9.3 instead...

comment:183 in reply to: ↑ 182 ; follow-up: Changed 2 years ago by leif

Replying to dimpase:

we can just blacklist gcc 4.9.2 and build our 4.9.3 instead...

or any Debian and Ubuntu GCC... ;-) [although someone just reported Sage's GCC doesn't build with some distro's 6.1.1, but that was on Arch Linux IIRC]

(Not the first time they cause trouble, and sometimes they won't even fix bugs / release versions with existing bugfixes.)

comment:184 follow-up: Changed 2 years ago by fbissey

There is also that bit earlier that looks strange to me

[fflas_ffpack-2.2.2] -----------------------------------------------
[fflas_ffpack-2.2.2] checking whether to use run time optimization... yes
[fflas_ffpack-2.2.2]   *** OPTIMIZATION ***  
[fflas_ffpack-2.2.2] checking best threshold for Strassen-Winograd matrix multiplication... see below
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::Modular<double> == 
[fflas_ffpack-2.2.2] problem
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::Modular<float> == 
[fflas_ffpack-2.2.2] problem
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::ModularBalanced<double> == 
[fflas_ffpack-2.2.2] problem
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::ModularBalanced<float> == 
[fflas_ffpack-2.2.2] problem

Something healthy should look like

checking whether to use run time optimization... yes
  *** OPTIMIZATION ***  
checking best threshold for Strassen-Winograd matrix multiplication... see below
  == Wino/BLAS threshold for Givaro::Modular<double> == 

fgemm 768x768: 0.0181467 s, 49.9248 Gffops
1Wino 768x768: 0.0225657 s, 40.148 Gffops

fgemm 1280x1280: 0.10682 s, 39.265 Gffops
1Wino 1280x1280: 0.128083 s, 32.7468 Gffops

fgemm 1792x1792: 0.249509 s, 46.1273 Gffops
1Wino 1792x1792: 0.23448 s, 49.0839 Gffops

fgemm 1792x1792: 0.217029 s, 53.0307 Gffops
1Wino 1792x1792: 0.306783 s, 37.5157 Gffops

fgemm 2304x2304: 0.449982 s, 54.3604 Gffops
1Wino 2304x2304: 0.521436 s, 46.9112 Gffops

fgemm 2816x2816: 0.838325 s, 53.274 Gffops
1Wino 2816x2816: 0.846912 s, 52.7338 Gffops

fgemm 3328x3328: 1.46935 s, 50.1711 Gffops
1Wino 3328x3328: 1.49322 s, 49.3692 Gffops

fgemm 3840x3840: 2.38059 s, 47.5706 Gffops
1Wino 3840x3840: 2.16039 s, 52.4193 Gffops

fgemm 3840x3840: 2.20834 s, 51.2812 Gffops
1Wino 3840x3840: 2.12749 s, 53.2299 Gffops

fgemm 3584x3584: 1.80012 s, 51.1484 Gffops
1Wino 3584x3584: 1.82305 s, 50.5051 Gffops

fgemm 3712x3712: 2.0441 s, 50.0441 Gffops
1Wino 3712x3712: 2.02077 s, 50.6217 Gffops

fgemm 3712x3712: 2.06589 s, 49.5161 Gffops
1Wino 3712x3712: 2.00375 s, 51.0516 Gffops
done (3712)
....

comment:185 follow-up: Changed 2 years ago by leif

P.S.: The most recent version of Sage I had built with GCC 4.9.2 is 6.10.beta6. :-)

Just pulled from here and started a (re)build (probably an upgrade adventure; I so far at least had to set LD_LIBRARY_PATH because sage-uncompress-spkg was replaced by a Python script...)

comment:186 in reply to: ↑ 185 ; follow-up: Changed 2 years ago by leif

Replying to leif:

P.S.: The most recent version of Sage I had built with GCC 4.9.2 is 6.10.beta6. :-)

Just pulled from here and started a (re)build (probably an upgrade adventure; I so far at least had to set LD_LIBRARY_PATH because sage-uncompress-spkg was replaced by a Python script...)

Indeed, with vanilla GCC 4.9.2, I'm getting the same errors for FFLAS-FFPACK.

But Debian should know that .3 and .4 are bugfix releases in the 4.9 series...

comment:187 in reply to: ↑ 186 Changed 2 years ago by fbissey

Replying to leif:

Replying to leif:

P.S.: The most recent version of Sage I had built with GCC 4.9.2 is 6.10.beta6. :-)

Just pulled from here and started a (re)build (probably an upgrade adventure; I so far at least had to set LD_LIBRARY_PATH because sage-uncompress-spkg was replaced by a Python script...)

Indeed, with vanilla GCC 4.9.2, I'm getting the same errors for FFLAS-FFPACK.

But Debian should know that .3 and .4 are bugfix releases in the 4.9 series...

It looks avx2 related to me. Does your test machine support that?

comment:188 in reply to: ↑ 183 ; follow-up: Changed 2 years ago by dimpase

Replying to leif:

Replying to dimpase:

we can just blacklist gcc 4.9.2 and build our 4.9.3 instead...

or any Debian and Ubuntu GCC... ;-) [although someone just reported Sage's GCC doesn't build with some distro's 6.1.1, but that was on Arch Linux IIRC]

Sage with the branch on this ticket does build on Arch with gcc 6.1.1 (that's what I have on my shiny new Thinkpad X1 Carbon (4th gen.) :-))

I did't try without this branch...

(Not the first time they cause trouble, and sometimes they won't even fix bugs / release versions with existing bugfixes.)

comment:189 in reply to: ↑ 184 Changed 2 years ago by leif

Replying to fbissey:

There is also that bit earlier that looks strange to me

[fflas_ffpack-2.2.2] -----------------------------------------------
[fflas_ffpack-2.2.2] checking whether to use run time optimization... yes
[fflas_ffpack-2.2.2]   *** OPTIMIZATION ***  
[fflas_ffpack-2.2.2] checking best threshold for Strassen-Winograd matrix multiplication... see below
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::Modular<double> == 
[fflas_ffpack-2.2.2] problem
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::Modular<float> == 
[fflas_ffpack-2.2.2] problem
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::ModularBalanced<double> == 
[fflas_ffpack-2.2.2] problem
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::ModularBalanced<float> == 
[fflas_ffpack-2.2.2] problem


That's the same error; in my config.log I have:

...
config.status:1478: executing libtool commands
configure:21501: checking whether to use run time optimization
configure:21518: result: yes
configure:21527: checking best threshold for Strassen-Winograd matrix multiplication
configure:21529: result: see below
configure:21552: g++-4.9 -o conftest -I. -I.. -I/tmp/sage-build-tmp/fflas_ffpack-2.2.2/src -I/tmp/sage-build-tmp/fflas_ffpack-2.2.2/src/fflas-ffpack -g -O2 -fPIC  -std=gnu++11  -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -fabi-version=6  -O2  -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -fPIC -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mfpmath=sse -fabi-version=6 -I/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/include -I/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local//include     -D__FFLASFFPACK_HAVE_CBLAS  -DFLTTYPE=Givaro::Modular<double> -DOPTIMISATION_MODE  -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib -Wl,-rpath,/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib  conftest.cpp  -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib -lcblas -latlas   -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local//lib -lgivaro -lgmpxx -lgmp   >&5
In file included from ./fflas-ffpack/fflas/fflas_simd.h:208:0,
                 from fflas-ffpack/fflas/fflas_freduce.h:33,
                 from fflas-ffpack/fflas/fflas.h:104,
                 from conftest.cpp:91:
./fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::sll128(Simd256i_base::vect_t)':
./fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: error: there are no arguments to '_mm256_bslli_epi128' that depend on a template parameter, so a declaration of '_mm256_bslli_epi128' must be available [-fpermissive]
  static INLINE CONST vect_t sll128(const vect_t a) { return _mm256_bslli_epi128(a, s); }
                                                                                     ^
./fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
./fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::srl128(Simd256i_base::vect_t)':
./fflas-ffpack/fflas/fflas_simd/simd256.inl:102:85: error: there are no arguments to '_mm256_bsrli_epi128' that depend on a template parameter, so a declaration of '_mm256_bsrli_epi128' must be available [-fpermissive]
  static INLINE CONST vect_t srl128(const vect_t a) { return _mm256_bsrli_epi128(a, s); }
                                                                                     ^
configure:21552: $? = 1
configure: program exited with status 1
configure: failed program was:
...
configure:21565: result: problem
configure:21588: g++-4.9 -o conftest -I. -I.. -I/tmp/sage-build-tmp/fflas_ffpack-2.2.2/src -I/tmp/sage-build-tmp/fflas_ffpack-2.2.2/src/fflas-ffpack -g -O2 -fPIC  -std=gnu++11  -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -fabi-version=6  -O2  -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -fPIC -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mfpmath=sse -fabi-version=6 -I/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/include -I/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local//include     -D__FFLASFFPACK_HAVE_CBLAS  -DFLTTYPE=Givaro::Modular<float> -DOPTIMISATION_MODE  -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib -Wl,-rpath,/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib  conftest.cpp  -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib -lcblas -latlas   -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local/lib -L/data/Sage/release/devel/sage-6.10.beta6-gcc-4.9.2/local//lib -lgivaro -lgmpxx -lgmp   >&5
In file included from ./fflas-ffpack/fflas/fflas_simd.h:208:0,
                 from fflas-ffpack/fflas/fflas_freduce.h:33,
                 from fflas-ffpack/fflas/fflas.h:104,
                 from conftest.cpp:91:
./fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::sll128(Simd256i_base::vect_t)':
./fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: error: there are no arguments to '_mm256_bslli_epi128' that depend on a template parameter, so a declaration of '_mm256_bslli_epi128' must be available [-fpermissive]
  static INLINE CONST vect_t sll128(const vect_t a) { return _mm256_bslli_epi128(a, s); }
                                                                                     ^
./fflas-ffpack/fflas/fflas_simd/simd256.inl:94:85: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
./fflas-ffpack/fflas/fflas_simd/simd256.inl: In static member function 'static Simd256i_base::vect_t Simd256i_base::srl128(Simd256i_base::vect_t)':
./fflas-ffpack/fflas/fflas_simd/simd256.inl:102:85: error: there are no arguments to '_mm256_bsrli_epi128' that depend on a template parameter, so a declaration of '_mm256_bsrli_epi128' must be available [-fpermissive]
  static INLINE CONST vect_t srl128(const vect_t a) { return _mm256_bsrli_epi128(a, s); }
                                                                                     ^
configure:21588: $? = 1
configure: program exited with status 1
configure: failed program was:
...
configure:21601: result: problem
...

and so on.

comment:190 follow-up: Changed 2 years ago by fbissey

Can you try with --disable-avx?

comment:191 in reply to: ↑ 188 Changed 2 years ago by leif

Replying to dimpase:

Replying to leif:

Replying to dimpase:

we can just blacklist gcc 4.9.2 and build our 4.9.3 instead...

or any Debian and Ubuntu GCC... ;-) [although someone just reported Sage's GCC doesn't build with some distro's 6.1.1, but that was on Arch Linux IIRC]

Sage with the branch on this ticket does build on Arch with gcc 6.1.1 (that's what I have on my shiny new Thinkpad X1 Carbon (4th gen.) :-))

Does it also build with SAGE_INSTALL_GCC=yes? (Which is what I meant, since that happens in case we'd blacklist [also newer] distro's GCCs.)

comment:192 in reply to: ↑ 190 Changed 2 years ago by leif

Replying to fbissey:

Can you try with --disable-avx?

With -fpermissive in CXXFLAGS I'm getting:

...
[fflas_ffpack-2.2.2] config.status: executing libtool commands
[fflas_ffpack-2.2.2] -----------------------------------------------
[fflas_ffpack-2.2.2] checking whether to use run time optimization... yes
[fflas_ffpack-2.2.2]   *** OPTIMIZATION ***  
[fflas_ffpack-2.2.2] checking best threshold for Strassen-Winograd matrix multiplication... see below
[fflas_ffpack-2.2.2]   == Wino/BLAS threshold for Givaro::Modular<double> == 
[fflas_ffpack-2.2.2] 
[fflas_ffpack-2.2.2] fgemm 768x768: 0.0556863 s, 16.2692 Gffops
[fflas_ffpack-2.2.2] 1Wino 768x768: 0.0571907 s, 15.8412 Gffops
[fflas_ffpack-2.2.2] 
[fflas_ffpack-2.2.2] fgemm 1280x1280: 0.246778 s, 16.9962 Gffops
[fflas_ffpack-2.2.2] 1Wino 1280x1280: 0.23861 s, 17.5781 Gffops
[fflas_ffpack-2.2.2] 
[fflas_ffpack-2.2.2] fgemm 1280x1280: 0.246413 s, 17.0214 Gffops
[fflas_ffpack-2.2.2] 1Wino 1280x1280: 0.239146 s, 17.5387 Gffops
[fflas_ffpack-2.2.2] 
[fflas_ffpack-2.2.2] fgemm 1024x1024: 0.127811 s, 16.802 Gffops
[fflas_ffpack-2.2.2] 1Wino 1024x1024: 0.12628 s, 17.0057 Gffops
[fflas_ffpack-2.2.2] 
[fflas_ffpack-2.2.2] fgemm 896x896: 0.0857503 s, 16.7771 Gffops
[fflas_ffpack-2.2.2] 1Wino 896x896: 0.083541 s, 17.2208 Gffops
[fflas_ffpack-2.2.2] done (896)
...
[fflas_ffpack-2.2.2] Successfully installed fflas_ffpack-2.2.2

comment:193 follow-up: Changed 2 years ago by fbissey

And for what I asked?

comment:194 in reply to: ↑ 193 Changed 2 years ago by leif

Replying to fbissey:

And for what I asked?

The machine supports AVX2 (and BMI/BMI2), but GCC 4.9 only supports the former.

Do you want me to retry without -fpermissive and configuring FFLAS-FFPACK --without-avx?

(The Sage library is still building...)

comment:195 Changed 2 years ago by leif

Did anyone (any buildbot) try with GCC 4.8 (we no longer blacklist)?

comment:196 follow-up: Changed 2 years ago by fbissey

Well that was what I was asking for, but if permissive work it is probably a better solution (performance wise at least).

As for gcc 4.8 that's my current default on power7 but there are other issues there.

comment:197 in reply to: ↑ 196 ; follow-ups: Changed 2 years ago by leif

Replying to fbissey:

Well that was what I was asking for, but if permissive work it is probably a better solution (performance wise at least).

Ok, I could push a patched branch, if no-one else does.

(Sage build/upgrade complete!)

Haven't run tests though, docbuilding in progress...


As for gcc 4.8 that's my current default on power7 but there are other issues there.

Well I could test with some vanilla GCC 4.8, but Ubuntu 14.04.x LTS has some 4.8.2-ubuntu-something, so probably a buildbot could as well.

comment:198 in reply to: ↑ 197 ; follow-up: Changed 2 years ago by leif

Replying to leif:

Replying to fbissey:

As for gcc 4.8 that's my current default on power7 but there are other issues there.

Well I could test with some vanilla GCC 4.8, but Ubuntu 14.04.x LTS has some 4.8.2-ubuntu-something, so probably a buildbot could as well.

I'm now also building on Ubuntu 14.04 x86_64, but that box doesn't have AVX, only SSE 4.1...

comment:199 in reply to: ↑ 197 ; follow-up: Changed 2 years ago by leif

Replying to leif:

Replying to fbissey:

Well that was what I was asking for, but if permissive work it is probably a better solution (performance wise at least).

Ok, I could push a patched branch, if no-one else does.

(Sage build/upgrade complete!)

Haven't run tests though, docbuilding in progress...

Besides some unrelated SageNB tests, all (long) tests passed.

comment:200 Changed 2 years ago by fbissey

OK. As good as expected. For the gcc-4.8 machine I don't expect problem, even without --permissive if the avx theory is correct.

comment:201 in reply to: ↑ 199 Changed 2 years ago by leif

Replying to leif:

Replying to leif:

Replying to fbissey:

Well that was what I was asking for, but if permissive work it is probably a better solution (performance wise at least).

Ok, I could push a patched branch, if no-one else does.

(Sage build/upgrade complete!)

Haven't run tests though, docbuilding in progress...

Besides some unrelated SageNB tests, all (long) tests passed.

(That refers to the build with GCC 4.9.2 and -fpermissive, on a mchine with AVX2.)

comment:202 in reply to: ↑ 198 ; follow-up: Changed 2 years ago by leif

Replying to leif:

Replying to leif:

Replying to fbissey:

As for gcc 4.8 that's my current default on power7 but there are other issues there.

Well I could test with some vanilla GCC 4.8, but Ubuntu 14.04.x LTS has some 4.8.2-ubuntu-something, so probably a buildbot could as well.

I'm now also building on Ubuntu 14.04 x86_64, but that box doesn't have AVX, only SSE 4.1...


Ok, with gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2 on Ubuntu 14.04 x86_64 (machine without AVX) ptestlong passed (modulo few known, unrelated issues).

Same with vanilla FSF GCC 4.8.5 on a machine with AVX2.

I didn't (have to) pass -fpermissive in either case, so that's apparently only needed for GCC 4.9 <= 4.9.2.

comment:203 in reply to: ↑ 202 Changed 2 years ago by leif

Replying to leif:

I didn't (have to) pass -fpermissive in either case, so that's apparently only needed for GCC 4.9 <= 4.9.2.

Note that Debian's (AFAIK), Ubuntu's, and some other distros' GCCs omit the patch level in gcc -dumpversion though; if we want to check the version, we have to use e.g. (the one from the first line of) $CXX --version instead.

comment:204 follow-up: Changed 2 years ago by leif

Random note:

If SAGE_FAT_BINARY=yes, we should only --disable-sse on x86, not x86_64, IMHO.

comment:205 in reply to: ↑ 204 ; follow-up: Changed 2 years ago by leif

Replying to leif:

If SAGE_FAT_BINARY=yes, we should only --disable-sse on x86, not x86_64, IMHO.

Clément, is it possible to disable just SSE 4.2 (or 4.x)?

comment:206 Changed 2 years ago by leif

Will push a new branch later, unless somebody else does (or Clément updates his):

  • build/pkgs/fflas_ffpack/spkg-install

    diff --git a/build/pkgs/fflas_ffpack/spkg-install b/build/pkgs/fflas_ffpack/spkg-install
    index d37385d..bd854b5 100755
    a b  
    11#!/usr/bin/env bash
    22
    33if [ -z "$SAGE_LOCAL" ]; then
    4     echo >&2 "SAGE_LOCAL undefined ... exiting"
     4    echo >&2 "Error: SAGE_LOCAL undefined ... exiting"
    55    echo >&2 "Maybe run 'sage --sh'?"
    66    exit 1
    77fi
    cp "$SAGE_ROOT"/config/config.* build-aux 
    2626export CFLAGS="-g -O2 -fPIC $CFLAGS"
    2727export CXXFLAGS="-g -O2 -fPIC $CXXFLAGS"
    2828
     29# GCC 4.9 <= 4.9.2 is broken, cf. https://trac.sagemath.org/ticket/17635#comment:178 ff.:
     30case `$CXX -dumpversion 2>/dev/null` in
     31    4.9.[0-2])
     32        CXXFLAGS+=" -fpermissive"
     33        ;;
     34    4.9) # some distros' GCCs omit the patch level
     35        if $CXX --version 2>/dev/null | head -n 1 | grep '4.9.[0-2]' >/dev/null; then
     36            CXXFLAGS+=" -fpermissive"
     37        fi
     38esac
     39
    2940if [ "$SAGE64" = "yes" ]; then
    3041   echo "64 bit build"
    3142   CFLAGS="-m64 $CFLAGS "; export CFLAGS
    fi 
    3748
    3849if [ "$LINBOX_BLAS" != "" ]; then
    3950    echo "Using environment variable LINBOX_BLAS=$LINBOX_BLAS"
    40 elif [ $UNAME = "CYGWIN" ]; then
     51elif [ "$UNAME" = "CYGWIN" ]; then
    4152    # TODO: we should install a suitable blas.pc on Cygwin
    4253    echo "Using system-wide Cygwin LAPACK BLAS."
    4354    if [ ! -f "/usr/lib/libblas.a" ]; then
    44         echo "*************************************************"
    45         echo "*"
    46         echo "* On Cygwin you must install the standard LAPACK Cygwin package"
    47         echo "* via the Cygwin setup.exe program in the 'Math' category."
    48         echo "*"
    49         echo "*************************************************"
     55        echo >&2 "*************************************************"
     56        echo >&2 "*"
     57        echo >&2 "* Error: On Cygwin you must install the standard LAPACK Cygwin package"
     58        echo >&2 "* via the Cygwin setup.exe program in the 'Math' category."
     59        echo >&2 "*"
     60        echo >&2 "*************************************************"
    5061        exit 1
    5162    fi
    5263    LINBOX_BLAS="-lblas"
    fi 
    7990    --enable-optimization --disable-static --enable-precompilation \
    8091    --disable-openmp $FFLAS_FFPACK_CONFIGURE
    8192if [ $? -ne 0 ]; then
    82     echo "Error configuring fflas-ffpack"
     93    echo >&2 "Error configuring fflas-ffpack"
    8394    exit 1
    8495fi
    8596
    8697$MAKE
    8798if [ $? -ne 0 ]; then
    88     echo "Error building fflas-ffpack"
     99    echo >&2 "Error building fflas-ffpack"
    89100    exit 1
    90101fi
    91102
    92103$MAKE install
    93104if [ $? -ne 0 ]; then
    94     echo "Error installing fflas-ffpack"
     105    echo >&2 "Error installing fflas-ffpack"
    95106    exit 1
    96107fi

comment:207 Changed 2 years ago by git

  • Commit changed from b7233c9cb64829f3c8a64b04aa744bdbb007ed27 to 7e8eb3a794d96c79900cd1567ccf325bd023c836

Branch pushed to git repo; I updated commit sha1. New commits:

7e8eb3aadd pacth fixing the gcc <= 4.9.2 issue by leif.

comment:208 in reply to: ↑ 205 ; follow-up: Changed 2 years ago by fbissey

Replying to leif:

Replying to leif:

If SAGE_FAT_BINARY=yes, we should only --disable-sse on x86, not x86_64, IMHO.

Clément, is it possible to disable just SSE 4.2 (or 4.x)?

Darn the configure.ac is so confusing and the macro folder so full of unused stuff that I got the configuration wrong. It looks like we just have --{en,dis}able-simd no granularity.


New commits:

7e8eb3aadd pacth fixing the gcc <= 4.9.2 issue by leif.

comment:209 Changed 2 years ago by fbissey

  • Status changed from needs_work to positive_review

looks good to me. Back to the bots please.

comment:210 in reply to: ↑ 208 ; follow-up: Changed 2 years ago by cpernet

Indeed, there is no granularity. If I understand correctly, you want to disable all sse>2 on x86 and all avx{1,2} on x86_64. There is no such option for the moment in fflas-ffpack. Given the fact that with a fat binary, the user will get a BLAS compiled for nehalem (very slow), I see almost no interest in terms of effciency to get the sse4 options turned on in fflas-ffpack when building for x86_64.

The mess in all the simd/sse checking macros comes partly from the fact that the standard macro ax_x86_features only work on x86 architectures.

comment:211 in reply to: ↑ 210 Changed 2 years ago by fbissey

Replying to cpernet:

The mess in all the simd/sse checking macros comes partly from the fact that the standard macro ax_x86_features only work on x86 architectures.

You are not exactly checking instructions for platforms besides x86(_64). Is it forward thinking?

comment:212 Changed 2 years ago by vbraun

  • Branch changed from u/cpernet/givaro_fflasffpack_linbox to 7e8eb3a794d96c79900cd1567ccf325bd023c836
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:213 Changed 2 years ago by leif

  • Commit 7e8eb3a794d96c79900cd1567ccf325bd023c836 deleted

Just for the record: At least on the Debian 8 64-bit buildbot, -fpermissive turned the previous errors into just warnings now.

http://build.sagedev.org/release/builders/%20%20slow%20AIMS%20%20%28Debian%208%2064%20bit%29%20incremental/builds/690/steps/compile_1/logs/stdio

comment:214 Changed 2 years ago by jdemeyer

Glad to see that this is finally merged!

comment:215 Changed 2 years ago by leif

There seems to be some issue with building the interface(s) when MAPLE is found:

...
checking for FLINT... found
checking for NTL >= 5.0... found
checking for SACLIB >= 2.0... not found
checking for MAPLE >= 9.0... ./configure: line 18187:
/bin/maple.system.type: No such file or directory
found
checking whether to compile the SAGE interface... yes
checking whether gnuplot works... yes
checking whether ps2pdf works... yes
...
Making all in interfaces
make[5]: Entering directory
'/home/dcoudert/sage-dev/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces'
Making all in driver
make[6]: Entering directory
'/home/dcoudert/sage-dev/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces/driver'
/bin/sh ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../..  -I../.. -I. -I../../linbox -DDISABLE_COMMENTATOR -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR
-I/home/dcoudert/sage-dev/sage/local/include -std=gnu++11  -g -fPIC -c -o lb-element.lo lb-element.C
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I../.. -I. -I../../linbox -DDISABLE_COMMENTATOR -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR -I/home/dcoudert/sage-dev/sage/local/include -std=gnu++11 -g -fPIC -c lb-element.C  -fPIC -DPIC -o .libs/lb-element.o
In file included from ./lb-domain-functor.h:30:0,
                 from ./lb-domain-function.h:30,
                 from ./lb-element-data.h:33,
                 from lb-element.C:30:
./lb-domain-type.h:31:33: fatal error: linbox/field/givaro..h: No such file or directory
#include "linbox/field/givaro.h"
                                ^
compilation terminated.
Makefile:571: recipe for target 'lb-element.lo' failed
make[6]: *** [lb-element.lo] Error 1
make[6]: Leaving directory
'/home/dcoudert/sage-dev/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces/driver'
Makefile:428: recipe for target 'all-recursive' failed
make[5]: *** [all-recursive] Error 1
make[5]: Leaving directory
'/home/dcoudert/sage-dev/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces'
Makefile:582: recipe for target 'all-recursive' failed
make[4]: *** [all-recursive] Error 1
make[4]: Leaving directory
'/home/dcoudert/sage-dev/sage/local/var/tmp/sage/build/linbox-1.4.2/src'
Makefile:453: recipe for target 'all' failed
make[3]: *** [all] Error 2
make[3]: Leaving directory
'/home/dcoudert/sage-dev/sage/local/var/tmp/sage/build/linbox-1.4.2/src'
Error building LinBox.

(Snippet from sage-devel report today.)

comment:216 follow-ups: Changed 2 years ago by chapoton

I have problems (can not build linbox) since the update, I am attaching the log. If somebody could have a look..

comment:217 in reply to: ↑ 216 Changed 2 years ago by leif

Replying to chapoton:

I have problems (can not build linbox) since the update, I am attaching the log. If somebody could have a look..

gcc version 4.9.3 (Ubuntu 4.9.3-8ubuntu2~14.04) 

FWIW, it builds with vanilla FSF GCC 4.9.3.

comment:218 in reply to: ↑ 216 Changed 2 years ago by leif

Replying to chapoton:

I have problems (can not build linbox) since the update, I am attaching the log. If somebody could have a look..

Could you please truncate the log to 833 lines? We don't need the log of N build attempts... ;-)

Changed 2 years ago by chapoton

shorter linbox log

comment:219 follow-up: Changed 2 years ago by leif

Probably due to a further (configure) issue with MAPLE:

checking for MAPLE >= 9.0... ./configure: line 18187: /bin/maple.system.type: No such file or directory
./configure: line 18198: /bin/maple: No such file or directory
cat: maple_version.txt: No such file or directory
./configure: line 18201: test: -lt: unary operator expected
found
./configure: line 18219: test: -ge: unary operator expected
checking whether to compile the SAGE interface... yes

This time there's no found though, but interestingly the header file linbox/field/givaro.h exists...

comment:220 follow-up: Changed 2 years ago by leif

Oh, there is a found -- between the syntax errors...

So for now disabling the MAPLE interface like Volker suggested may make sense.

comment:221 in reply to: ↑ 220 Changed 2 years ago by leif

Replying to leif:

So for now disabling the MAPLE interface like Volker suggested may make sense.

  • build/pkgs/linbox/spkg-install

    diff --git a/build/pkgs/linbox/spkg-install b/build/pkgs/linbox/spkg-install
    index b051d4c..ecd6c25 100755
    a b if [ "$SAGE_FAT_BINARY" = yes ]; then 
    6565    LINBOX_CONFIGURE="--without-ocl $LINBOX_CONFIGURE"
    6666fi
    6767
     68# Building MAPLE interface appears to be broken:
     69# http://trac.sagemath.org/ticket/17635#comment:215 ff. (*after* LinBox ticket got merged!)
     70LINBOX_CONFIGURE="--without-maple $LINBOX_CONFIGURE"
     71
    6872# We disable openmp because of build failures, see
    6973# http://trac.sagemath.org/ticket/17635#comment:67
    7074./configure --prefix="$SAGE_LOCAL" --libdir="$SAGE_LOCAL/lib" \

HTH, not tested.

comment:222 follow-up: Changed 2 years ago by leif

I don't have MAPLE installed, but regardless of whether I pass --without-maple, --with-maple=no, or nothing (related), I'm always getting

checking for MAPLE >= 9.0... not found

so I'm not sure if that option works at all; --with-maple=no is said to be the default anyway:

  --with-maple=<path>|yes|no
                          Use Maple library. If argument is no, you do not
                          have the library installed on your machine (set as
                          default). If argument is yes or <empty> that means
                          the library is well installed and so reachable.
                          Otherwise you give the <path> to the directory which
                          contains the Software.

comment:223 in reply to: ↑ 222 Changed 2 years ago by leif

Replying to leif:

I don't have MAPLE installed, but regardless of whether I pass --without-maple, --with-maple=no, or nothing (related), I'm always getting

checking for MAPLE >= 9.0... not found

so I'm not sure if that option works at all...

Looking at configure, it (--with-maple=no at least) should work (checking and not found will always appear).

Changed 2 years ago by dcoudert

linbox log when trying the trick of comment https://trac.sagemath.org/ticket/17635#comment:221

comment:224 follow-up: Changed 2 years ago by dcoudert

unfortunately not working :(

comment:225 in reply to: ↑ 224 Changed 2 years ago by leif

Replying to dcoudert:

unfortunately not working :(

Can you retry with --with-maple=no instead? (As said, that should work, despite the "checking" message.)

comment:226 follow-up: Changed 2 years ago by leif

Hmmm, LinBox doesn't find David's M4RIE...


The checking for MAPLE regardless of --with-maple=no is by the way due to Checking all external packages ... (--with-all=yes).

comment:227 follow-up: Changed 2 years ago by dcoudert

I tried with --with-maple=no and got the same result.

...
checking for MAPLE >= 9.0... ./configure: line 18187: /bin/maple.system.type: No such file or directory
found
...
./lb-domain-type.h:31:33: fatal error: linbox/field/givaro.h: No such file or directory
...

Concerning m4rie, it is installed (sorry, configured in french)

%m:%S%~%s> yum install m4rie
Modules complémentaires chargés : langpacks
Le paquet m4rie-20140914-1.fc21.x86_64 est déjà installé dans sa dernière version
Rien à faire

comment:228 in reply to: ↑ 226 Changed 2 years ago by fbissey

Replying to leif:

Hmmm, LinBox doesn't find David's M4RIE...

linbox has plenty of useless check (stuff that it is not using "yet") that's one of them. Because of this m4rie is not a dependency of linbox and yes sometimes it won't be found.

comment:229 in reply to: ↑ 227 Changed 2 years ago by fbissey

Replying to dcoudert:

I tried with --with-maple=no and got the same result.

...
checking for MAPLE >= 9.0... ./configure: line 18187: /bin/maple.system.type: No such file or directory
found
...
./lb-domain-type.h:31:33: fatal error: linbox/field/givaro.h: No such file or directory
...

Concerning m4rie, it is installed (sorry, configured in french)

%m:%S%~%s> yum install m4rie
Modules complémentaires chargés : langpacks
Le paquet m4rie-20140914-1.fc21.x86_64 est déjà installé dans sa dernière version
Rien à faire

C'est pas grave parce qu'on veux la version de sage pas celle de Fedora.

Are you working for sage-on-fedora?

comment:230 in reply to: ↑ 219 Changed 2 years ago by fbissey

Replying to leif:

This time there's no found though, but interestingly the header file linbox/field/givaro.h exists...

That's very interesting because I don't have it. Is it generated somehow if you have maple?

comment:231 Changed 2 years ago by leif

I have a patch (rather a hack, for Sage only):

  • new file uild/pkgs/linbox/patches/dont_search__usr__usr_local.patch

    diff --git a/build/pkgs/linbox/patches/dont_search__usr__usr_local.patch b/build/pkgs/linbox/patches/dont_search__usr__usr_local.patch
    new file mode 100644
    index 0000000..98c4ab4
    - +  
     1Remove /usr and /usr/local from LinBox's "default checking path"
     2such that we can configure with '--with-default="$SAGE_LOCAL"' and
     3no "external" packages from elsewhere get picked up (such as MAPLE,
     4whose interface appears to be broken / break the build at the moment).
     5
     6Also, remove "unknown" from MAPLE_HOME_PATH as otherwise still /usr
     7and /usr/local will get searched for MAPLE, even though no longer
     8part of DEFAULT_CHECKING_PATH.
     9
     10
     11--- linbox-1.4.2/configure
     12+++ linbox-1.4.2/configure
     13@@ -16579,8 +16579,8 @@
     14                       echo "Default path = /usr /usr/local "
     15                       DEFAULT_CHECKING_PATH="/usr /usr/local "
     16             else
     17-                      echo "Default path = $withval /usr /usr/local "
     18-                      DEFAULT_CHECKING_PATH="$withval /usr /usr/local "
     19+                      echo "Default path = $withval "
     20+                      DEFAULT_CHECKING_PATH="$withval "
     21            fi
     22 
     23 else
     24@@ -16617,7 +16617,7 @@
     25     FFLAS_FFPACK_HOME_PATH="${DEFAULT_CHECKING_PATH}"
     26       LIDIA_HOME_PATH="${DEFAULT_CHECKING_PATH}"
     27       SACLIB_HOME_PATH="${DEFAULT_CHECKING_PATH}"
     28-      MAPLE_HOME_PATH="${DEFAULT_CHECKING_PATH} unknown"
     29+      MAPLE_HOME_PATH="${DEFAULT_CHECKING_PATH}"
     30       BLAS_HOME_PATH="${DEFAULT_CHECKING_PATH}"
     31 fi
     32 
  • build/pkgs/linbox/spkg-install

    diff --git a/build/pkgs/linbox/spkg-install b/build/pkgs/linbox/spkg-install
    index b051d4c..a06ae07 100755
    a b if [ "$SAGE_FAT_BINARY" = yes ]; then 
    6565    LINBOX_CONFIGURE="--without-ocl $LINBOX_CONFIGURE"
    6666fi
    6767
     68# Building MAPLE interface appears to be broken:
     69# http://trac.sagemath.org/ticket/17635#comment:215 ff. (*after* LinBox ticket got merged!)
     70# LINBOX_CONFIGURE="--without-maple $LINBOX_CONFIGURE"
     71LINBOX_CONFIGURE="--with-maple=no $LINBOX_CONFIGURE"
     72# To make this work, currently a patch to 'configure' is needed though:
     73# patches/dont_search__usr__usr_local.patch
     74
    6875# We disable openmp because of build failures, see
    6976# http://trac.sagemath.org/ticket/17635#comment:67
    7077./configure --prefix="$SAGE_LOCAL" --libdir="$SAGE_LOCAL/lib" \
    71             --with-all="$SAGE_LOCAL" \
     78            --with-all="$SAGE_LOCAL" --with-default="$SAGE_LOCAL" \
    7279            --enable-sage --disable-static --disable-openmp \
    7380            $LINBOX_CONFIGURE
    7481if [ $? -ne 0 ]; then

comment:232 follow-up: Changed 2 years ago by dcoudert

I don't know how to test this hack (not able to locate file linbox-1.4.2/configure).

comment:233 in reply to: ↑ 232 ; follow-ups: Changed 2 years ago by dimpase

Replying to dcoudert:

I don't know how to test this hack (not able to locate file linbox-1.4.2/configure).

you don't need this file physically; one does monkey-patching, as done in many Sage packages; create a directory patches/ in build/pkgs/linbox/, and put the patch above there. Then you also need to add a "patch block" to spkg-install somewhere near the top of spkg-install:

    # Apply all patches
    for patch in patches/*.patch; do
        [ -r "$patch" ] || continue  # Skip non-existing or non-readable patches
        echo "Applying $patch"
        patch -p1 <"$patch"
        if [ $? -ne 0 ]; then
            echo >&2 "Error applying '$patch'"
            return 1
        fi
    done
}

I wonder why there is no default functionality for this, for this block is in dozens of spkg-install files...

comment:234 Changed 2 years ago by dimpase

Should one open a new ticket for this?

comment:235 Changed 2 years ago by vbraun

Definitely new ticket!!!

comment:236 in reply to: ↑ 233 ; follow-up: Changed 2 years ago by dcoudert

I unsuccessfully tried. I'm missing something in copy/paste, etc. I will wait for the new ticket and a git branch. Safer for me. Thanks, David.

Replying to dimpase:

Replying to dcoudert:

I don't know how to test this hack (not able to locate file linbox-1.4.2/configure).

you don't need this file physically; one does monkey-patching, as done in many Sage packages; create a directory patches/ in build/pkgs/linbox/, and put the patch above there. Then you also need to add a "patch block" to spkg-install somewhere near the top of spkg-install:

...

comment:237 in reply to: ↑ 236 Changed 2 years ago by dimpase

Replying to dcoudert:

I unsuccessfully tried. I'm missing something in copy/paste, etc. I will wait for the new ticket and a git branch. Safer for me.

we continue on this on #21482.

comment:238 in reply to: ↑ 233 Changed 2 years ago by leif

Replying to dimpase:

Replying to dcoudert:

I don't know how to test this hack (not able to locate file linbox-1.4.2/configure).

Sorry, I thought it was obvious, although I could in addition have attached the patch (to simply be applied from $SAGE_ROOT), but forgot.


you don't need this file physically; one does monkey-patching, as done in many Sage packages; create a directory patches/ in build/pkgs/linbox/, and put the patch above there.

No, it's one patch which

  • modifies LinBox's spkg-install (the "patch loop" is already there by the way)
  • creates a patch to LinBox's configure which will be applied (by spkg-install) when reinstalling LinBox.

In other words, cd $SAGE_ROOT && patch -p1 < inline_patch_from_the_comment_above.patch, then ./sage -f linbox && make.


Then you also need to add a "patch block" to spkg-install somewhere near the top of spkg-install:

    # Apply all patches
    for patch in patches/*.patch; do
        [ -r "$patch" ] || continue  # Skip non-existing or non-readable patches
        echo "Applying $patch"
        patch -p1 <"$patch"
        if [ $? -ne 0 ]; then
            echo >&2 "Error applying '$patch'"
            return 1
        fi
    done
}

I wonder why there is no default functionality for this, for this block is in dozens of spkg-install files...

Erik B. has a ticket for this, don't know the number right know (probably search for "apply-patches").

comment:239 follow-up: Changed 2 years ago by embray

This seems to have broken my Cygwin build in 7.4beta5 (at least). It fails during build of sagelib itself, linking one of the Cython modules:

$ g++ -shared -Wl,--enable-auto-image-base -L/home/embray/src/sagemath/sage/local/lib -Wl,-rpath,/home/embray/src/sagemath/sage/local/lib -L/home/embray/src/sagemath/sage/local/lib -Wl,-rpath,/home/embray/src/sagemath/sage/local/lib build/temp.cygwin-2.5.1-x86_64-2.7/home/embray/src/sagemath/sage/src/build/cythonized/sage/libs/linbox/linbox.o -L/home/embray/src/sagemath/sage/local/lib -L/home/embray/src/sagemath/sage/local//lib -L/home/embray/src/sagemath/sage/local/lib/python2.7/config -L/home/embray/src/sagemath/sage/local/lib -lfflas -llinboxsage -lffpack -lblas -liml -llinbox -lntl -lgivaro -lflint -lflint -lfplll -lmpfr -lgmp -lgmp -lgmpxx -lstdc++ -lpython2.7 -o build/lib.cygwin-2.5.1-x86_64-2.7/sage/libs/linbox/linbox.dll
/home/embray/src/sagemath/sage/local/lib/liblinboxsage.a(linbox-sage.o): In function `double* linbox_modn_dense_matrix_matrix_multiply<double>(double, double*, double*, double*, unsigned long, unsigned long, unsigned long)':
/home/embray/src/sagemath/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces/sage/linbox-sage.C:313: undefined reference to `Givaro::Modular<double, double>::Element_ptr FFLAS::fgemm<Givaro::Modular<double, double> >(Givaro::Modular<double, double> const&, FFLAS::FFLAS_TRANSPOSE, FFLAS::FFLAS_TRANSPOSE, unsigned long, unsigned long, unsigned long, Givaro::Modular<double, double>::Element, Givaro::Modular<double, double>::ConstElement_ptr, unsigned long, Givaro::Modular<double, double>::ConstElement_ptr, unsigned long, Givaro::Modular<double, double>::Element, Givaro::Modular<double, double>::Element_ptr, unsigned long)'
/home/embray/src/sagemath/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces/sage/linbox-sage.C:313:(.text+0x35d9): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `Givaro::Modular<double, double>::Element_ptr FFLAS::fgemm<Givaro::Modular<double, double> >(Givaro::Modular<double, double> const&, FFLAS::FFLAS_TRANSPOSE, FFLAS::FFLAS_TRANSPOSE, unsigned long, unsigned long, unsigned long, Givaro::Modular<double, double>::Element, Givaro::Modular<double, double>::ConstElement_ptr, unsigned long, Givaro::Modular<double, double>::ConstElement_ptr, unsigned long, Givaro::Modular<double, double>::Element, Givaro::Modular<double, double>::Element_ptr, unsigned long)'
/home/embray/src/sagemath/sage/local/lib/liblinboxsage.a(linbox-sage.o): In function `float* linbox_modn_dense_matrix_matrix_multiply<float>(float, float*, float*, float*, unsigned long, unsigned long, unsigned long)':
/home/embray/src/sagemath/sage/local/var/tmp/sage/build/linbox-1.4.2/src/interfaces/sage/linbox-sage.C:313: undefined reference to `Givaro::Modular<float, float>::Element_ptr FFLAS::fgemm<Givaro::Modular<float, float> >(Givaro::Modular<float, float> const&, FFLAS::FFLAS_TRANSPOSE, FFLAS::FFLAS_TRANSPOSE, unsigned long, unsigned long, unsigned long, Givaro::Modular<float, float>::Element, Givaro::Modular<float, float>::ConstElement_ptr, unsigned long, Givaro::Modular<float, float>::ConstElement_ptr, unsigned long, Givaro::Modular<float, float>::Element, Givaro::Modular<float, float>::Element_ptr, unsigned long)'
...

followed by dozens of similar errors. There's nothing strange in the build logs for any of fflas-ffpack, givaro, or linbox themselves. GCC is 5.4.0. Any ideas? I'm guessing maybe linker flag order, which can be more sensitive on Windows--not sure though.

Last edited 2 years ago by embray (previous) (diff)

comment:240 in reply to: ↑ 239 ; follow-up: Changed 2 years ago by embray

Replying to embray:

I'm guessing maybe linker flag order, which can be more sensitive on Windows--not sure though.

Yep, in fact switching the order to -llinboxsage -lfflas seems to have fixed it.

comment:241 Changed 2 years ago by embray

Seemingly also related to this ticket is this mess

gcc -I/usr/include/ncurses -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/cysignals -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local/include/python2.7 -I/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/numpy/core/include -I/home/embray/src/sagemath/sage/src -I/home/embray/src/sagemath/sage/src/sage/ext -I/home/embray/src/sagemath/sage/src/build/cythonized -I/home/embray/src/sagemath/sage/src/build/cythonized/sage/ext -I/home/embray/src/sagemath/sage/local/include/python2.7 -c /home/embray/src/sagemath/sage/src/build/cythonized/sage/matrix/matrix_modn_dense_double.cpp -o build/temp.cygwin-2.5.1-x86_64-2.7/home/embray/src/sagemath/sage/src/build/cythonized/sage/matrix/matrix_modn_dense_double.o -O2 -Wall -g -DNDEBUG -U_LB_DEBUG -DDISABLE_COMMENTATOR -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mbmi -mbmi2 -fabi-version=6 -DFFLAS_COMPILED -DFFPACK_COMPILED -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mbmi -mbmi2 -mfpmath=sse -fabi-version=6 -I/home/embray/src/sagemath/sage/local/include/linbox -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local//include -I/home/embray/src/sagemath/sage/local/include -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mbmi -mbmi2 -mfpmath=sse -fabi-version=6 -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local//include -I/home/embray/src/sagemath/sage/local/include -O2 -Wall -DNDEBUG -UFFLASFFPACK_DEBUG -D__FFLASFFPACK_HAVE_CBLAS -g -O2 -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mbmi -mbmi2 -fabi-version=6 -DFFLAS_COMPILED -DFFPACK_COMPILED -std=gnu++11 -mmmx -mpopcnt -msse -msse2 -msse3 -msse4.1 -msse4.2 -mavx -mavx2 -mfma -mbmi -mbmi2 -mfpmath=sse -fabi-version=6 -I/home/embray/src/sagemath/sage/local/include -I/home/embray/src/sagemath/sage/local//include -I/home/embray/src/sagemath/sage/local/include -D_XPG6 -fno-strict-aliasing
In file included from /home/embray/src/sagemath/sage/local/include/flint/flint.h:46:0,
                 from /home/embray/src/sagemath/sage/src/build/cythonized/sage/matrix/matrix_modn_dense_double.cpp:497:
/home/embray/src/sagemath/sage/local/include/flint/longlong.h:45:0: warning: "add_ssaaaa" redefined
 #define add_ssaaaa(sh, sl, ah, al, bh, bl)                 \
 ^
In file included from /home/embray/src/sagemath/sage/local/include/recint/recdefine.h:145:0,
                 from /home/embray/src/sagemath/sage/local/include/recint/ruruint.h:43,
                 from /home/embray/src/sagemath/sage/local/include/recint/rumanip.h:41,
                 from /home/embray/src/sagemath/sage/local/include/recint/ruconvert.h:43,
                 from /home/embray/src/sagemath/sage/local/include/gmp++/gmp++_int.h:26,
                 from /home/embray/src/sagemath/sage/local/include/gmp++/gmp++.h:56,
                 from /home/embray/src/sagemath/sage/local/include/givaro/givinteger.h:19,
                 from /home/embray/src/sagemath/sage/local/include/givaro/modular-int8.h:29,
                 from /home/embray/src/sagemath/sage/local/include/givaro/modular.h:21,
                 from /home/embray/src/sagemath/sage/src/build/cythonized/sage/matrix/matrix_modn_dense_double.cpp:490:
/home/embray/src/sagemath/sage/local/include/recint/reclonglong.h:1906:0: note: this is the location of the previous definition
 #define add_ssaaaa(sh, sl, ah, al, bh, bl) \
 ^
In file included from /home/embray/src/sagemath/sage/local/include/flint/flint.h:46:0,
                 from /home/embray/src/sagemath/sage/src/build/cythonized/sage/matrix/matrix_modn_dense_double.cpp:497:

...

In file included from /home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas_igemm/igemm_kernels.h:98:0,
                 from /home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas_igemm/igemm.h:45,
                 from /home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas_fgemm/fgemm_classical.inl:43,
                 from /home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas_fgemm.inl:440,
                 from /home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas.h:109,
                 from /home/embray/src/sagemath/sage/local/include/fflas-ffpack/ffpack/ffpack.h:46,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/matrixdomain/blas-matrix-domain.h:45,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/matrix-domain.h:68,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/transpose-matrix.h:47,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/matrixdomain/matrix-domain.inl:34,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/matrixdomain/matrix-domain.h:1233,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/densematrix/blas-matrix.h:55,
                 from /home/embray/src/sagemath/sage/local/include/linbox/matrix/dense-matrix.h:79,
                 from /home/embray/src/sagemath/sage/src/build/cythonized/sage/matrix/matrix_modn_dense_double.cpp:491:
/home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas_igemm/igemm_kernels.inl:154:4: error: ‘B1’ was not declared in this scope
    B1 = simd::load( blB+1*StepB);
    ^
/home/embray/src/sagemath/sage/local/include/fflas-ffpack/fflas/fflas_igemm/igemm_kernels.inl:155:27: error: no matching function for call to ‘Simd256_impl<true, true, true, 8>::fmaddxin(vect_t&, vect_t&, int)’
    simd::fmaddxin(C0,A0,B0);
                           ^

with many similar errors following.

Last edited 2 years ago by embray (previous) (diff)

comment:242 follow-up: Changed 2 years ago by embray

It seems to me that most of the problems I'm having are due to the reliance on #distutils: flags in the Cython, which are not being collated intelligently enough...

comment:243 in reply to: ↑ 240 Changed 2 years ago by jdemeyer

Replying to embray:

Replying to embray:

I'm guessing maybe linker flag order, which can be more sensitive on Windows--not sure though.

Yep, in fact switching the order to -llinboxsage -lfflas seems to have fixed it.

See the library_order_list variable in src/module_list.py

comment:244 in reply to: ↑ 242 Changed 2 years ago by jdemeyer

Replying to embray:

It seems to me that most of the problems I'm having are due to the reliance on #distutils: flags in the Cython, which are not being collated intelligently enough...

If you have more details, I could try to help with that.

comment:245 follow-up: Changed 2 years ago by jdemeyer

Just to be clear: you are using the same version of Cython as vanilla Sage, right?

comment:246 follow-up: Changed 2 years ago by embray

I'm not sure it's coming from library_order_list. For example linbox.cpp contains:

        "libraries": [
            "fflas",
            "linboxsage",
            "ffpack",
            "blas",
            "iml",
            "gmpxx",
            "linbox",
            "flint",
            "ntl",
            "fplll",
            "gmp",
            "givaro",
            "mpfr",
            "flint",
            "gmp"
        ],

This does not reflect what's in library_order_list at all.

comment:247 in reply to: ↑ 245 Changed 2 years ago by embray

Replying to jdemeyer:

Just to be clear: you are using the same version of Cython as vanilla Sage, right?

Yes, this is after a fresh build following make maintainer-clean.

comment:248 in reply to: ↑ 246 Changed 2 years ago by jdemeyer

Replying to embray:

I'm not sure it's coming from library_order_list. For example linbox.cpp contains:

        "libraries": [
            "fflas",
            "linboxsage",
            "ffpack",
            "blas",
            "iml",
            "gmpxx",
            "linbox",
            "flint",
            "ntl",
            "fplll",
            "gmp",
            "givaro",
            "mpfr",
            "flint",
            "gmp"
        ],

This does not reflect what's in library_order_list at all.

That's because the library order is changed after that metadata is written to the .cpp file. If you want to check the real library order, look at the gcc command line.

comment:250 Changed 2 years ago by embray

If you look above you'll see that what's in the list I quoted above is what's in the gcc command line, so no the library order is not being changed.

Actually I'm wrong, it's not exactly the same. But the order it's ending up in is not the same as library_order_list would give either.

Last edited 2 years ago by embray (previous) (diff)

comment:251 Changed 2 years ago by embray

In https://github.com/cython/cython/pull/412 you wrote

The whole point of this pull request is that I don't want to list all libraries in setup.py. That is a mess to maintain.

I completely agree there. But I don't know if using the metadata in Cython files is the best way either. That works fine for projects where you have a few small files. And I agree the json metadata is useful for debugging. But ultimately I think defining hand-written Extension instances is the most explicit and understandable way to go. That doesn't mean they all have to be dumped into one big file as in module_list.py.

Other projects manage this differently. For example Numpy has a "setup.py" for each subpackage that defines build instructions for modules in that package. Similarly subpackages in Astropy can have a setup_package.py for information about how to build that package, which is reasonably well documented. Those are just two examples, not saying that Sage needs to follow suit. Just that I don't know that Cython can always be smart enough here on its own.

Anyways, I'm straying off topic.

comment:252 Changed 2 years ago by embray

Here's an example of one of the more non-trivial setup_package.py's from astropy: https://github.com/astropy/astropy/blob/master/astropy/_erfa/setup_package.py

Another feature I added to it not long ago is pre-/post-command hooks, so each subpackage can perform their own special steps before or after specific ./setup.py commands. This would be useful, for example, for the stuff in sage_setup.autogen--autogenerated the needed Python modules at the appropriate build step; no reliance on a makefile.

comment:253 follow-up: Changed 2 years ago by embray

Anyways, adding fflas_ffpack_libraries to library_order_list in the appropriate spot seems to have fixed all my woes for now so I will open a new ticket with that fix--thanks jdemeyer for the tip-off on that even if I was confused by it at first.

I spoke too soon--my build is still just "cythonizing" the files; it didn't compile them yet.

I will open a separate ticket for further discussion on how Sage manages its extension configurations.

Last edited 2 years ago by embray (previous) (diff)

comment:254 in reply to: ↑ 253 ; follow-up: Changed 2 years ago by embray

Replying to embray:

Anyways, adding fflas_ffpack_libraries to library_order_list in the appropriate spot seems to have fixed all my woes for now so I will open a new ticket with that fix--thanks jdemeyer for the tip-off on that even if I was confused by it at first.

I spoke too soon--my build is still just "cythonizing" the files; it didn't compile them yet.

Updating library_order_list fixed my first issue (with linking sage.libs.linbox.linbox) but not the second issue.

comment:255 in reply to: ↑ 254 Changed 2 years ago by jdemeyer

Replying to embray:

Updating library_order_list fixed my first issue

Maybe you can create a ticket just with this fix.

Note: See TracTickets for help on using tickets.