update fplll to version 5.x

[malb]

Once fplll 5.0.0 is out we should upgrade:

• upgrade the SPKG
• add fpylll
• drop the current custom fplll interface

See ​https://groups.google.com/forum/#!topic/sage-devel/v_g81diO2pU for discussion

SPKGS

• fplll ​https://github.com/fplll/fplll/releases/download/5.0.3rc0/fplll-5.0.3rc0.tar.gz
• fpylll ​https://github.com/fplll/fpylll/releases/download/0.2.2dev/fpylll-0.2.2dev.tar.gz

[malb]

New commits:

​d646b8b	fplll 5.0.1 & drop sage.libs.fplll
​e34070f	add fpylll
​7b0a9f1	A.LLL() and A.BKZ() using fpylll

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​debb5a1	fix doctests
​d3df341	formatting

[malb]

Everything should be in place now. Waiting for ​https://github.com/fplll/fplll/pull/189 being merged to cut a new release. Then this ticket should be good to go.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7bd03e0

fplll 5.0.2

[malb]

Should be good to go now.

[jpflori]

Changes look good to me. But fplllfails to build on a linux/POWER7:

[fplll-5.0.2] In file included from bkz_param.cpp:4:0:
[fplll-5.0.2] io/json.hpp: In constructor 'nlohmann::basic_json<ObjectType, Arra
yType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFlo
atType, AllocatorType>::basic_json() [with ObjectType = std::map; ArrayType = st
d::vector; StringType = std::basic_string<char>; BooleanType = bool; NumberInteg
erType = long int; NumberUnsignedType = long unsigned int; NumberFloatType = dou
ble; AllocatorType = std::allocator]':
[fplll-5.0.2] io/json.hpp:1062:5: error: conversion from 'nlohmann::basic_json<>
::value_t' to non-scalar type 'nlohmann::basic_json<>::type_data_t' requested
[fplll-5.0.2]      basic_json() = default;
[fplll-5.0.2]      ^
[fplll-5.0.2] bkz_param.cpp: In function 'std::vector<fplll::Strategy> fplll::lo
ad_strategies_json(const string&)':
[fplll-5.0.2] bkz_param.cpp:79:8: note: synthesized method 'nlohmann::basic_json
<ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsign
edType, NumberFloatType, AllocatorType>::basic_json() [with ObjectType = std::ma
p; ArrayType = std::vector; StringType = std::basic_string<char>; BooleanType = 
bool; NumberIntegerType = long int; NumberUnsignedType = long unsigned int; Numb
erFloatType = double; AllocatorType = std::allocator]' first required here
[fplll-5.0.2]    json js;

[jpflori]

Looks like:

• ​https://github.com/nlohmann/json/issues/259#issuecomment-223776640

[jpflori]

I'm indeed using GCC 4.8, but as reported there it also make clang choke and a more recent version of json probably makes the problem disappear as the involved types have been suppressed/modified.

[malb]

Thanks for tracking it down. Can you try with the new json.hpp from ​https://github.com/nlohmann/json/blob/develop/src/json.hpp so check if it fixes your problem? I'd then update on my end.

Just dump json.hpp in fplll/fplll/io.

[jpflori]

With the latest json.hpp it just errors out without trying saying that the compiler (GCC 4.8) is unsupported. Nice.

[jpflori]

FYI, forcing json.hpp to give GCC 4.8.3 a try, it compiles seemingly fine.

[malb]

Hi, so to clarify: if we update json.hpp and drop its GCC 4.8 check then it seems to compile on your linux/POWER7 box?

[jpflori]

Yes exactly.

[jpflori]

Just note it does not check for 4.8 but for >= 4.9.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​83a78f0

fpylll depends on sagelib

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f986fe1

update to new fplll/fpylll upstream releases

[malb]

Updated to new upstream versions, now supporting GCC 4.8. Thanks.

[jpflori]

(I fixed the links in the ticket description.)

It seems the new Linbox from #17635 (closed) fails to build on top of this:

[linbox-1.4.2]  #include <fplll/dpe.h>

[jpflori]

CC'ing Clément for the Linbox/FPLLL interaction.

[jpflori]

At the moment we can just add a Linbox patch here to include from the nr subdir.

[jpflori]

Unfortunately, that's not enough:

[linbox-1.4.2] ../../linbox/algorithms/lattice.inl: In function 'void LinBox::ll
lReduceInBase(LinBox::BlasMatrix<_Field>&, LinBox::BlasMatrix<_Field>&, const Li
nBox::latticeMethod::latticeFPLLL&)':
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:342:35: error: 'class fplll::
ZZ_mat<__mpz_struct [1]>' has no member named 'Set'
[linbox-1.4.2]                                  B.Set((int)i,(int)j,x);
[linbox-1.4.2]                                    ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:349:5: error: 'proved' is not
 a member of 'fplll'
[linbox-1.4.2]      fplll::proved<ZT,double> lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:349:21: error: expected prima
ry-expression before ',' token  
[linbox-1.4.2]      fplll::proved<ZT,double> lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]                      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:349:22: error: expected prima
ry-expression before 'double'   
[linbox-1.4.2]      fplll::proved<ZT,double> lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]                       ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:349:22: error: expected ';' b
efore 'double'
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:351:5: error: 'lllMethod' was not declared in this scope
[linbox-1.4.2]      lllMethod.LLL();
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:356:5: error: 'wrapper' is not a member of 'fplll'
...
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:351:5: error: 'lllMethod' was not declared in this scope
[linbox-1.4.2]      lllMethod.LLL();
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:356:5: error: 'wrapper' is not a member of 'fplll'
[linbox-1.4.2]      fplll::wrapper lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:356:20: error: expected ';' before 'lllMethod'
[linbox-1.4.2]      fplll::wrapper lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]                     ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:358:5: error: 'lllMethod' was not declared in this scope
[linbox-1.4.2]      lllMethod.LLL();
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:363:5: error: 'heuristic' is not a member of 'fplll'
[linbox-1.4.2]      fplll::heuristic<ZT,double> lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:363:24: error: expected primary-expression before ',' token  
[linbox-1.4.2]      fplll::heuristic<ZT,double> lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]                         ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:363:25: error: expected primary-expression before 'double'   
[linbox-1.4.2]      fplll::heuristic<ZT,double> lllMethod(&B,meth.getPrecision(),
[linbox-1.4.2]                          ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:363:25: error: expected ';' before 'double'
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:366:5: error: 'lllMethod' was not declared in this scope
[linbox-1.4.2]      lllMethod.LLL();
[linbox-1.4.2]      ^
[linbox-1.4.2] ../../linbox/algorithms/lattice.inl:376:23: error: 'class fplll::ZZ_mat<__mpz_struct [1]>' has no member named 'Get'
[linbox-1.4.2]      H.setEntry(i,j, B.Get(i,j) );

[jpflori]

If fixing this really requires a lot of work in Linbox, we could just disable fplll in Linbox waiting for a Linbox fix... if that's an option.

[cpernet]

Replying to jpflori:

we could just disable fplll in Linbox waiting for a Linbox fix... if that's an option.

This sounds like the most reasonable choice. I don't think any linbox dependency on fplll is actually exposed to sage at the moment. The hook was kept for the future.

[jpflori]

With this strategy everything looks ok. Let the patchbots (or Volker?) do their testing jobs.

---

Last 10 new commits:

​98cc68c	Only try relevant extension for OS when loading libSingular.
​399f051	Add cddlib as a dependency and make gfanlib building explicit
​797c539	Improve the cleaning
​75e383a	use Singular.pc almost everywhere
​a0f55ea	Singular.pc is now used everywhere. plural.pyx doesn't need givaro or readline or m anymore.
​55447da	Do not build (a second time) libfactory when building Singular.
​e5c10ae	Cleaner removal of files.
​29077a7	Use wildcards for singular and letterplace in module_list.py.
​f366df4	Merge branch 'singular4' into fplll5
​cc8a95f	Let Linbox build by disabling its fplll linking.

[jpflori]

Oops, it seems I pushed a wrong branch. I'll do a force push in a few seconds, sorry about that...

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. This was a forced push. New commits:

​2dc1fd8	Merge remote-tracking branch 'trac/develop' into fplll
​3f2d20c	Let Linbox build by disabling its fplll linking.

[jpflori]

Sorry for the mess, here's a clean branch where I only merged trac/develop and disabled fplll linking wihtin linbox.

[tmonteil]

I started the tests before the last changes of Jean-Pierre, but i got doctests failure on 32bit system, which i guess will still be here after applying the last two commits too.

sage -t --long --warn-long 124.3 src/sage/modules/free_module_integer.py
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 78, in sage.modules.free_module_integer.IntegerLattice
Failed example:
    IntegerLattice(A)
Expected:
    Free module of degree 10 and rank 10 over Integer Ring
    User basis matrix:
    [ 0  1  2  0  1  2 -1  0 -1 -1]
    [ 0  1  0 -3  0  0  0  0  3 -1]
    [ 1  1 -1  0 -3  0  0  1  2 -2]
    [-1  2 -1 -1  2 -2  1 -1  0 -1]
    [ 1  0 -4  2  0  1 -2 -1  0  0]
    [ 2  3  0  1  1  0 -2  3  0  0]
    [-2 -3 -2  0  0  1 -1  1  3 -2]
    [-3  0 -1  0 -2 -1 -2  1 -1  1]
    [ 1  4 -1  1  2  2  1  0  3  1]
    [-1 -1  0 -3 -1  2  2  3 -1  0]
Got:
    Free module of degree 10 and rank 10 over Integer Ring
    User basis matrix:
    [ 0  1  2  0  1  2 -1  0 -1 -1]
    [ 1 -2  1  1 -2  2 -1  1  0  1]
    [ 0  2 -2  2 -1 -2  1  0 -1 -2]
    [ 0  1  0 -3  0  0  0  0  3 -1]
    [ 1  0 -4  2  0  1 -2 -1  0  0]
    [ 2  3  0  1  1  0 -2  3  0  0]
    [-3 -2 -3  2  2 -1  0  0  0 -2]
    [-3  0 -1  0 -2 -1 -2  1 -1  1]
    [ 1  4 -1  1  2  2  1  0  3  1]
    [-1 -1  0 -3 -1  2  2  3 -1  0]
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 95, in sage.modules.free_module_integer.IntegerLattice
Failed example:
    sage.crypto.gen_lattice(type='modular', m=10, seed=42, dual=True, lattice=True)
Expected:
    Free module of degree 10 and rank 10 over Integer Ring
    User basis matrix:
    [ 0  1  2  0  1  2 -1  0 -1 -1]
    [ 0  1  0 -3  0  0  0  0  3 -1]
    [ 1  1 -1  0 -3  0  0  1  2 -2]
    [-1  2 -1 -1  2 -2  1 -1  0 -1]
    [ 1  0 -4  2  0  1 -2 -1  0  0]
    [ 2  3  0  1  1  0 -2  3  0  0]
    [-2 -3 -2  0  0  1 -1  1  3 -2]
    [-3  0 -1  0 -2 -1 -2  1 -1  1]
    [ 1  4 -1  1  2  2  1  0  3  1]
    [-1 -1  0 -3 -1  2  2  3 -1  0]
Got:
    Free module of degree 10 and rank 10 over Integer Ring
    User basis matrix:
    [ 0  1  2  0  1  2 -1  0 -1 -1]
    [ 1 -2  1  1 -2  2 -1  1  0  1]
    [ 0  2 -2  2 -1 -2  1  0 -1 -2]
    [ 0  1  0 -3  0  0  0  0  3 -1]
    [ 1  0 -4  2  0  1 -2 -1  0  0]
    [ 2  3  0  1  1  0 -2  3  0  0]
    [-3 -2 -3  2  2 -1  0  0  0 -2]
    [-3  0 -1  0 -2 -1 -2  1 -1  1]
    [ 1  4 -1  1  2  2  1  0  3  1]
    [-1 -1  0 -3 -1  2  2  3 -1  0]
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 203, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer
Failed example:
    L = IntegerLattice(sage.crypto.gen_lattice(type='modular', m=10, seed=42, dual=True)); L
Expected:
    Free module of degree 10 and rank 10 over Integer Ring
    User basis matrix:
    [ 0  1  2  0  1  2 -1  0 -1 -1]
    [ 0  1  0 -3  0  0  0  0  3 -1]
    [ 1  1 -1  0 -3  0  0  1  2 -2]
    [-1  2 -1 -1  2 -2  1 -1  0 -1]
    [ 1  0 -4  2  0  1 -2 -1  0  0]
    [ 2  3  0  1  1  0 -2  3  0  0]
    [-2 -3 -2  0  0  1 -1  1  3 -2]
    [-3  0 -1  0 -2 -1 -2  1 -1  1]
    [ 1  4 -1  1  2  2  1  0  3  1]
    [-1 -1  0 -3 -1  2  2  3 -1  0]
Got:
    Free module of degree 10 and rank 10 over Integer Ring
    User basis matrix:
    [ 0  1  2  0  1  2 -1  0 -1 -1]
    [ 1 -2  1  1 -2  2 -1  1  0  1]
    [ 0  2 -2  2 -1 -2  1  0 -1 -2]
    [ 0  1  0 -3  0  0  0  0  3 -1]
    [ 1  0 -4  2  0  1 -2 -1  0  0]
    [ 2  3  0  1  1  0 -2  3  0  0]
    [-3 -2 -3  2  2 -1  0  0  0 -2]
    [-3  0 -1  0 -2 -1 -2  1 -1  1]
    [ 1  4 -1  1  2  2  1  0  3  1]
    [-1 -1  0 -3 -1  2  2  3 -1  0]
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 513, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ
Failed example:
    L.reduced_basis[0]
Expected:
    (-1, 0, 0, 0, 0, 0, 1, 0, -1, 0, 0, 0, 0, 0, -2, 0, 0, 0, 0, 0,
     -1, 1, 0, 1, 1, 0, 0, 0, 3, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0)
Got:
    (1, 0, 0, 0, 0, -1, 1, 1, 0, 0, 1, -1, -1, -1, 0, 0, 0, 0, 0, -1, 0, 0, 1, 0, 0, 0, 0, 0, -1, 0, -1, 1, 1, -1, 1, 2, 0, 0, -1, -1)
**********************************************************************

Concerning src/sage/matrix/matrix_integer_dense.pyx i have a Timed out (and interrupt failed) without further explanations, i will run the tests by hand one by one.

I put it in "need info" not "need work" because i have to test the very last commit on a fresh install to confirm the issue, this might take some time since the VM is a bit slow.

[malb]

Looks like you're getting a different LLL reduced basis. Mhh, no idea why 32 bit systems might be different, we might have to wait to see what the patchbots report.

[jpflori]

Will the patchbots do anything if this is not positive_review?

[tmonteil]

On my fresh retry, i got the same failures (but no problem for src/sage/matrix/matrix_integer_dense.pyx which were probably due to some optional pkgs).

Replying to malb:

Looks like you're getting a different LLL reduced basis. Mhh, no idea why 32 bit systems might be different, we might have to wait to see what the patchbots report.

I do not know either, especially since LLL is deterministic, so perhaps is there some non-constant-length variable in the floating-point part of the code ?

Usually, the arch is sufficient to separate such behaviours (e.g. on my 64bit daily laptop, the tests are OK), and we can make specific tests for them.

But before settling down a #32bit doctest with those values, we should check that the bases are indeed LLL-reduced (it could be an explicit part of the test by the way).

For double checking, note that apart from my patchbot (actually a qemulated Pentium III), arando is the other running 32bit arch, which is run by Jeroen, so we could ask him to tell that patchbot to focus on that ticket.

Replying to jpflori:

Will the patchbots do anything if this is not positive_review?

Yes, since the patchbots are more a tool to help reviewing than to help releasing (for which the buildbot helps). Actually, being in needs_review gives you more chance to get some patchbots looking at your code (so let me put the ticket to needs_review, just to motivate some of those): ​https://github.com/sagemath/sage-patchbot/blob/master/sage_patchbot/patchbot.py#L420 Note that you can force the re-test with the ?kick trick : ​https://patchbot.sagemath.org/ticket/21221/?kick (this is why there is retry: True now, i clicked on that link yesterday).

[malb]

I can reproduce it in a 32-bit virtual machine:

sage: from sage.modules.free_module_integer import IntegerLattice
sage: A = sage.crypto.gen_lattice(type='modular', m=10, seed=42, dual=True)
sage: copy(A).LLL(algorithm="fpLLL:proved", fp="rr")[3]
(-1, 2, -1, -1, 2, -2, 1, -1, 0, -1)
sage: copy(A).LLL(algorithm="fpLLL:proved", fp="fp")[3]
(0, 1, 0, -3, 0, 0, 0, 0, 3, -1)

On a 64-bit system:

sage: copy(A).LLL(algorithm="fpLLL:proved", fp="fp")[3]
(-1, 2, -1, -1, 2, -2, 1, -1, 0, -1)
sage: copy(A).LLL(algorithm="fpLLL:proved", fp="rr")[3]
(-1, 2, -1, -1, 2, -2, 1, -1, 0, -1)

Note that the last outputs differ. In either case the output is LLL reduced (I'm not in favour of adding that test to the doctest though, since LLL-reducedness is not the main point there, thus it would be a bit distracting).

A cheap workaround is to simply pick a different seed for the doctest.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c4fa878

unify 32-bit and 64-bit doctest outputs

[malb]

This seems to have done the trick.

[jpflori]

Maybe we should still document that even with the same seed, the LLL reduction, although deterministic, might give different results on 32 and 64 platforms (different roundng for floats?) ?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1a15357

add warning about different LLL answers

[jpflori]

Good to go for me.

[fbissey]

Waiting for Volker's run report. But in sage-on-gentoo, I couldn't get the test suite for fpylll to run successfully when it was built using sagelib. I decided to hard disable it until I figure if it's broken on my side only or it is for everyone. Actually the fpylll tarball that is linked in this ticket doesn't include the test suite or does it?

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​d194317

fpylll 0.2.2dev

[malb]

Pushed fpylll 0.2.2dev which ships the testsuite and doesn't rely on environment variables.

[fbissey]

I am not sure that was fair to push that particular upgrade while Volker has already started to test stuff on the bot.

At least the change is minimal and fpylll doesn't currently have a spkg-check nor do we have py.test in the "vanilla" sage tree to run it. But I do, so I have concerns because it looks to me that fpylll has had its test suite run without sage with python 2.7 and 3.5 but it looks like its test suite when built against sage is sage testsuite itself. Just for fun

>>> Test phase: dev-python/fpylll-0.2.1
 * python2_7: running distutils-r1_run_phase python_test
=============================================================================== test session starts ===============================================================================
platform linux2 -- Python 2.7.10, pytest-2.8.7, py-1.4.30, pluggy-0.3.1 -- /usr/bin/python2.7
cachedir: .cache
rootdir: /scratch2/portage/dev-python/fpylll-0.2.1/work/fpylll-0.2.1dev, inifile: 
collecting ... collected 17 items / 1 errors

tests/test_bkz_python.py::test_bkz_init FAILED
tests/test_bkz_python.py::test_simple_bkz_call FAILED
tests/test_bkz_python.py::test_bkz_call FAILED
tests/test_gso.py::test_gso_init /scratch2/portage/dev-python/fpylll-0.2.1/temp/environment: line 2590:  5851 Segmentation fault      py.test -v
 * python3_4: running distutils-r1_run_phase python_test
=============================================================================== test session starts ===============================================================================
platform linux -- Python 3.4.3, pytest-2.8.7, py-1.4.30, pluggy-0.3.1 -- /usr/bin/python3.4
cachedir: .cache
rootdir: /scratch2/portage/dev-python/fpylll-0.2.1/work/fpylll-0.2.1dev, inifile: 
collecting ... collected 19 items

tests/test_bkz.py::test_bkz_init PASSED
tests/test_bkz.py::test_bkz_bkz PASSED
tests/test_bkz_python.py::test_bkz_init PASSED
tests/test_bkz_python.py::test_simple_bkz_call PASSED
tests/test_bkz_python.py::test_bkz_call PASSED
tests/test_gso.py::test_gso_init PASSED
tests/test_gso.py::test_gso_d PASSED
tests/test_gso.py::test_gso_int_gram_enabled PASSED
tests/test_gso.py::test_gso_update_gso PASSED
tests/test_lll.py::test_lll_init PASSED
tests/test_lll.py::test_lll_lll PASSED
tests/test_numpy.py::test_dump_mu PASSED
tests/test_numpy.py::test_dump_r PASSED
tests/test_precision.py::test_precision PASSED
tests/test_pruner.py::test_pruner_gso PASSED
tests/test_pruner.py::test_pruner_vec PASSED
tests/test_random.py::test_randomize PASSED
tests/test_simple_bkz.py::test_simple_bkz_init PASSED
tests/test_simple_bkz.py::test_simple_bkz_reduction PASSED

=========================================================================== 19 passed in 59.30 seconds ============================================================================

python3.4 results to show what happens when not built against sage.

I am not sure how SAGERUNTIME is generated but fpylll needs to be in it. From the look of it, it is generated so it is probably OK.

fpylll needs to be build against sagelib and sagelib needs fpylll to run. From my packaging point of view that means that

• I will have to split the building of the documentation in its own package, again. Otherwise I have circular dependencies that cannot be satisfied
• I can forget about ever achieving a state where I can take a package for sage and do build, test and install in that order. From now on there is no hope for testing sage before it is fully installed. Looking around it seems very common in python packages, it looks like testing before installing on the same machine is old school.

[fbissey]

Replying to fbissey:

• I will have to split the building of the documentation in its own package, again. Otherwise I have circular dependencies that cannot be satisfied

OK I won't have to split for now. Documentation builds fine without fpylll installed. Should future documentation make plots that depends directly or indirectly from fpylll and the split will have to happen.

[vbraun]

I'm getting some failures on OSX:

sage -t --long src/sage/matrix/matrix_integer_dense.pyx
**********************************************************************
File "src/sage/matrix/matrix_integer_dense.pyx", line 2726, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.?
Failed example:
    A.BKZ()
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.matrix.matrix_integer_dense.Matrix_integer_dense.?[1]>", line 1, in <module>
        A.BKZ()
      File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723)
        BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds),
      File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192)
      File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457)
    SignalError: Segmentation fault
**********************************************************************
File "src/sage/matrix/matrix_integer_dense.pyx", line 2732, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.?
Failed example:
    A.BKZ(use_givens=True)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.matrix.matrix_integer_dense.Matrix_integer_dense.?[3]>", line 1, in <module>
        A.BKZ(use_givens=True)
      File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723)
        BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds),
      File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192)
      File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457)
    SignalError: Segmentation fault
**********************************************************************
File "src/sage/matrix/matrix_integer_dense.pyx", line 2738, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.?
Failed example:
    A.BKZ(fp="fp")
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.matrix.matrix_integer_dense.Matrix_integer_dense.?[5]>", line 1, in <module>
        A.BKZ(fp="fp")
      File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723)
        BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds),
      File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192)
      File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457)
    SignalError: Segmentation fault
**********************************************************************
1 item had failures:
   3 of  67 in sage.matrix.matrix_integer_dense.Matrix_integer_dense.?
    [573 tests, 3 failures, 21.83 s]

and

sage -t --long src/sage/modules/free_module_integer.py
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 216, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer
Failed example:
    L.shortest_vector()
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer[2]>", line 1, in <module>
        L.shortest_vector()
      File "sage/misc/cachefunc.pyx", line 2038, in sage.misc.cachefunc.CachedMethodCaller.__call__ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:10183)
        w = self._instance_call(*args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1914, in sage.misc.cachefunc.CachedMethodCaller._instance_call (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:9647)
        return self.f(self._instance, *args, **kwds)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/modules/free_module_integer.py", line 633, in shortest_vector
        w = vector(ZZ, SVP.shortest_vector(L, *args, **kwds))
      File "src/fpylll/fplll/svpcvp.pyx", line 71, in fpylll.fplll.svpcvp.shortest_vector (build/src/fpylll/fplll/svpcvp.cpp:3122)
      File "src/cysignals/signals.pyx", line 102, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1273)
    RuntimeError: Aborted
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 454, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ
Failed example:
    L.BKZ(block_size=10)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ[6]>", line 1, in <module>
        L.BKZ(block_size=Integer(10))
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/modules/free_module_integer.py", line 466, in BKZ
        basis = [v for v in basis.BKZ(*args, **kwds) if v]
      File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723)
        BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds),
      File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192)
      File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457)
    SignalError: Segmentation fault
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 457, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ
Failed example:
    min(v.norm().n() for v in L.reduced_basis)
Expected:
    4.12310562561766
Got:
    5.19615242270663
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 510, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ
Failed example:
    L.HKZ()
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ[2]>", line 1, in <module>
        L.HKZ()
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/modules/free_module_integer.py", line 516, in HKZ
        return self.BKZ(block_size=self.rank())
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/modules/free_module_integer.py", line 466, in BKZ
        basis = [v for v in basis.BKZ(*args, **kwds) if v]
      File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723)
        BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds),
      File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192)
      File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457)
    SignalError: Segmentation fault
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 513, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ
Failed example:
    L.reduced_basis[0]
Expected:
    (0, 0, -1, -1, 0, 0, -1, 1, 0, 0, -1, 1, 1, 0, 0, 1, 1, 1, -1, 0, 0, 1, -1, 0, 0, -1, 0, 0, 1, 0, 0, -1, 0, 0, 0, 1, 1, 0, 0, -2)
Got:
    (-1, 0, 0, 2, 0, 0, 0, 0, -1, 1, 0, -1, 0, 1, 1, 0, 0, -2, 0, 1, 1, 1, 0, -1, 1, -1, -1, 0, -1, -1, -1, 1, -1, 0, 0, 0, 0, 0, 0, 0)
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 605, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.shortest_vector
Failed example:
    L.shortest_vector().norm().n()
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.shortest_vector[4]>", line 1, in <module>
        L.shortest_vector().norm().n()
      File "sage/misc/cachefunc.pyx", line 2038, in sage.misc.cachefunc.CachedMethodCaller.__call__ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:10183)
        w = self._instance_call(*args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1914, in sage.misc.cachefunc.CachedMethodCaller._instance_call (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:9647)
        return self.f(self._instance, *args, **kwds)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/modules/free_module_integer.py", line 633, in shortest_vector
        w = vector(ZZ, SVP.shortest_vector(L, *args, **kwds))
      File "src/fpylll/fplll/svpcvp.pyx", line 71, in fpylll.fplll.svpcvp.shortest_vector (build/src/fpylll/fplll/svpcvp.cpp:3122)
      File "src/cysignals/signals.pyx", line 102, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1273)
    RuntimeError: Aborted
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 660, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis
Failed example:
    v = L.shortest_vector(update_reduced_basis=False)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis[4]>", line 1, in <module>
        v = L.shortest_vector(update_reduced_basis=False)
      File "sage/misc/cachefunc.pyx", line 2038, in sage.misc.cachefunc.CachedMethodCaller.__call__ (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:10183)
        w = self._instance_call(*args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1914, in sage.misc.cachefunc.CachedMethodCaller._instance_call (/Users/buildslave-sage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:9647)
        return self.f(self._instance, *args, **kwds)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/modules/free_module_integer.py", line 633, in shortest_vector
        w = vector(ZZ, SVP.shortest_vector(L, *args, **kwds))
      File "src/fpylll/fplll/svpcvp.pyx", line 71, in fpylll.fplll.svpcvp.shortest_vector (build/src/fpylll/fplll/svpcvp.cpp:3122)
      File "src/cysignals/signals.pyx", line 102, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1273)
    RuntimeError: Aborted
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 661, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis
Failed example:
    L.update_reduced_basis(v)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 498, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 861, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis[5]>", line 1, in <module>
        L.update_reduced_basis(v)
    NameError: name 'v' is not defined
**********************************************************************
File "src/sage/modules/free_module_integer.py", line 662, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis
Failed example:
    bool(L.reduced_basis[0].norm() < B[0].norm())
Expected:
    True
Got:
    False
**********************************************************************
5 items had failures:
   1 of   4 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer
   2 of   9 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ
   2 of   5 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ
   1 of  11 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.shortest_vector
   3 of   8 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis
    [97 tests, 9 failures, 63.26 s]

[malb]

Replying to fbissey:

I am not sure that was fair to push that particular upgrade while Volker has already started to test stuff on the bot.

Sorry, I didn't mean to create more work, rather less.

[malb]

Mhh, wo we still have SSH access to an OS X box. It's hard to tell from here what may have caused it. I'll also ask on our fplll-devel mailinglist if people have OSX boxes.

[malb]

Looking at the failures, they look like the kind of failures I used to get. Was there maybe a mixup somewhere and an old branch was tested?

[fbissey]

I may try a build now that I finished my clang exploration.

[malb]

Hi there, @fbissey did you end up building this? @vbraun, anything I should do to trigger a re-test?

[fbissey]

Replying to malb:

Hi there, @fbissey did you end up building this? @vbraun, anything I should do to trigger a re-test?

I did a try twice. Each time I ended up with a mountain of doctests failing for some apparently unrelated reason. I decided to try again from a pristine tree but I haven't got time to do another attempt yet.

[vbraun]

I can try again ...

[vbraun]

Its definitely still happening on OSX...

[malb]

Thanks & sorry for the noise. I'll dig up an OS X box and debug there.

[malb]

This is strange: I built & tested on an OS X box now and all tests passed. Using

• branch: public/fplll5
• fplll: ​https://github.com/fplll/fplll/releases/download/5.0.3rc0/fplll-5.0.3rc0.tar.gz
• fpylll: ​https://github.com/fplll/fpylll/releases/download/0.2.2dev/fpylll-0.2.2dev.tar.gz

Not sure what to do about this now. Any suggestions?

[jpflori]

Apart from getting access to the problematic machine, no...

[malb]

On a second OSX box I'm now seeing:

ImportError: dlopen(/Users/vagrant/fplll/lib/python2.7/site-packages/fpylll/fplll/enumeration.so, 2): Symbol not found: __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd

Referenced from: /Users/vagrant/fplll/lib/python2.7/site-packages/fpylll/fplll/enumeration.so

Expected in: flat namespace
 in /Users/vagrant/fplll/lib/python2.7/site-packages/fpylll/fplll/enumeration.so

Any OSX experts out there who can make sense of this? More details at ​https://github.com/fplll/fplll/issues/205

[malb]

It seems this is caused by some bad interaction with brew. Starting from a system without homebrew and installing everything in Sage instead, I don't run into this problem. Running doctests now.

[malb]

Doctests pass, so I still cannot reproduce the buildbot failues.

[jpflori]

I'm currently building on the build bot and will try to reproduce.

[jpflori]

So on the OS X buildbot, the current branch is ok as far as the two files mentioned by Volker are cocnerned. I'll now merge with develop to see what changes.

[jpflori]

Hum, just merging trac/develop and issuing make, the tests still pass... Now trying to make distclean and make.

[jpflori]

Ok I can now reproduce it. Maybe openblas related?

[malb]

To reproduce, I'd merge trac/develop, make distclean and make? I'll give that a try.

[malb]

fplll doesn't use any blas, btw, so that shouldn't be it.

[jpflori]

Yes indeed, but I was looking for an easy culprit. And it has nothing to do with the problem here. Doing make distclean; configure --with-blas=atlas; make leads to the same errors.

[jpflori]

Wait, I don't see the same errors as Volker, at least with the latest build. It seems fpylll got built before sagelib and the interface to sage ismissing. Strange.

[jpflori]

(And I'm using the correct branch where sagelib is in the dependencies file for fpylll.)

[jpflori]

Forcing the recompilation of fpylll solved my issues... no more errors...

[malb]

Mhhh, sagelib is not the right thing to put as a dependency?

[jpflori]

It seems you're right. Putting sagelib in dependencies turns into inst_sagelib in build/make/Makefile and this target does not exist. The creation process of that Makefile should be modified. Or the field in the dependency file...

[jpflori]

Or put | $(SAGERUNTIME) in dependencies.

[vbraun]

The conway_polynomials package depends on sage runtime if you want an example.

[jpflori]

And maybe don't put it as an order only dependency in case fpylll must be recompiled if anything in the integer matrix class changes.

[jpflori]

But SAGERUNTIME is a phony target so it surely does not matter.

[jpflori]

With a properly built Sage I get no segfault. So maybe Volker setup ended up in broken state.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​848f211

add dependency on SAGERUNTIME

[jdemeyer]

Breakage: #21675