#21221 closed enhancement (fixed)
update fplll to version 5.x
Reported by:  malb  Owned by:  

Priority:  major  Milestone:  sage7.4 
Component:  packages: standard  Keywords:  
Cc:  tmonteil, jpflori, cpernet, jdemeyer  Merged in:  
Authors:  Martin Albrecht  Reviewers:  JeanPierre Flori, Thierry Monteil 
Report Upstream:  N/A  Work issues:  
Branch:  848f211 (Commits, GitHub, GitLab)  Commit:  
Dependencies:  #17635  Stopgaps: 
Description (last modified by )
Once fplll 5.0.0 is out we should upgrade:
 upgrade the SPKG
 add fpylll
 drop the current custom fplll interface
See https://groups.google.com/forum/#!topic/sagedevel/v_g81diO2pU for discussion
SPKGS
Change History (84)
comment:1 Changed 6 years ago by
 Cc tmonteil added
comment:2 Changed 6 years ago by
 Cc jpflori added
comment:3 Changed 6 years ago by
 Branch set to u/malb/t21221fplll5
 Commit set to 7b0a9f1d0b0870c6e855d2109264a8b7e90e1baf
comment:4 Changed 6 years ago by
 Description modified (diff)
comment:5 Changed 6 years ago by
 Commit changed from 7b0a9f1d0b0870c6e855d2109264a8b7e90e1baf to d3df34187889563414c98d405681c51ca6cd1035
comment:6 Changed 6 years ago by
Everything should be in place now. Waiting for https://github.com/fplll/fplll/pull/189 being merged to cut a new release. Then this ticket should be good to go.
comment:7 Changed 6 years ago by
 Commit changed from d3df34187889563414c98d405681c51ca6cd1035 to 7bd03e0a1d843df29f1c71654a3bd1a39410921e
Branch pushed to git repo; I updated commit sha1. New commits:
7bd03e0  fplll 5.0.2

comment:8 Changed 6 years ago by
 Description modified (diff)
 Status changed from new to needs_review
Should be good to go now.
comment:9 Changed 6 years ago by
Changes look good to me.
But fplll
fails to build on a linux/POWER7:
[fplll5.0.2] In file included from bkz_param.cpp:4:0: [fplll5.0.2] io/json.hpp: In constructor 'nlohmann::basic_json<ObjectType, Arra yType, StringType, BooleanType, NumberIntegerType, NumberUnsignedType, NumberFlo atType, AllocatorType>::basic_json() [with ObjectType = std::map; ArrayType = st d::vector; StringType = std::basic_string<char>; BooleanType = bool; NumberInteg erType = long int; NumberUnsignedType = long unsigned int; NumberFloatType = dou ble; AllocatorType = std::allocator]': [fplll5.0.2] io/json.hpp:1062:5: error: conversion from 'nlohmann::basic_json<> ::value_t' to nonscalar type 'nlohmann::basic_json<>::type_data_t' requested [fplll5.0.2] basic_json() = default; [fplll5.0.2] ^ [fplll5.0.2] bkz_param.cpp: In function 'std::vector<fplll::Strategy> fplll::lo ad_strategies_json(const string&)': [fplll5.0.2] bkz_param.cpp:79:8: note: synthesized method 'nlohmann::basic_json <ObjectType, ArrayType, StringType, BooleanType, NumberIntegerType, NumberUnsign edType, NumberFloatType, AllocatorType>::basic_json() [with ObjectType = std::ma p; ArrayType = std::vector; StringType = std::basic_string<char>; BooleanType = bool; NumberIntegerType = long int; NumberUnsignedType = long unsigned int; Numb erFloatType = double; AllocatorType = std::allocator]' first required here [fplll5.0.2] json js;
comment:10 Changed 6 years ago by
comment:11 Changed 6 years ago by
 Milestone set to sage7.4
 Status changed from needs_review to needs_work
I'm indeed using GCC 4.8, but as reported there it also make clang choke and a more recent version of json probably makes the problem disappear as the involved types have been suppressed/modified.
comment:12 Changed 6 years ago by
Thanks for tracking it down. Can you try with the new json.hpp
from https://github.com/nlohmann/json/blob/develop/src/json.hpp so check if it fixes your problem? I'd then update on my end.
comment:13 Changed 6 years ago by
With the latest json.hpp
it just errors out without trying saying that the compiler (GCC 4.8) is unsupported. Nice.
comment:14 Changed 6 years ago by
FYI, forcing json.hpp
to give GCC 4.8.3 a try, it compiles seemingly fine.
comment:15 Changed 6 years ago by
Hi, so to clarify: if we update json.hpp
and drop its GCC 4.8 check then it seems to compile on your linux/POWER7 box?
comment:16 Changed 6 years ago by
Yes exactly.
comment:17 Changed 6 years ago by
Just note it does not check for 4.8 but for >= 4.9.
comment:18 Changed 6 years ago by
 Commit changed from 7bd03e0a1d843df29f1c71654a3bd1a39410921e to 83a78f0ea53991c5ada939e50a95b0530d4bc8d4
Branch pushed to git repo; I updated commit sha1. New commits:
83a78f0  fpylll depends on sagelib

comment:19 Changed 6 years ago by
 Commit changed from 83a78f0ea53991c5ada939e50a95b0530d4bc8d4 to f986fe12472ccd88c79a7691cf2235fe4ca2bb0b
Branch pushed to git repo; I updated commit sha1. New commits:
f986fe1  update to new fplll/fpylll upstream releases

comment:20 Changed 6 years ago by
 Description modified (diff)
 Status changed from needs_work to needs_review
comment:21 Changed 6 years ago by
Updated to new upstream versions, now supporting GCC 4.8. Thanks.
comment:22 Changed 6 years ago by
 Description modified (diff)
 Reviewers set to JeanPierre Flori
(I fixed the links in the ticket description.)
It seems the new Linbox from #17635 (closed) fails to build on top of this:
[linbox1.4.2] #include <fplll/dpe.h>
comment:24 Changed 6 years ago by
At the moment we can just add a Linbox patch here to include from the nr
subdir.
comment:25 Changed 6 years ago by
Unfortunately, that's not enough:
[linbox1.4.2] ../../linbox/algorithms/lattice.inl: In function 'void LinBox::ll lReduceInBase(LinBox::BlasMatrix<_Field>&, LinBox::BlasMatrix<_Field>&, const Li nBox::latticeMethod::latticeFPLLL&)': [linbox1.4.2] ../../linbox/algorithms/lattice.inl:342:35: error: 'class fplll:: ZZ_mat<__mpz_struct [1]>' has no member named 'Set' [linbox1.4.2] B.Set((int)i,(int)j,x); [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:349:5: error: 'proved' is not a member of 'fplll' [linbox1.4.2] fplll::proved<ZT,double> lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:349:21: error: expected prima ryexpression before ',' token [linbox1.4.2] fplll::proved<ZT,double> lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:349:22: error: expected prima ryexpression before 'double' [linbox1.4.2] fplll::proved<ZT,double> lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:349:22: error: expected ';' b efore 'double' [linbox1.4.2] ../../linbox/algorithms/lattice.inl:351:5: error: 'lllMethod' was not declared in this scope [linbox1.4.2] lllMethod.LLL(); [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:356:5: error: 'wrapper' is not a member of 'fplll' ... [linbox1.4.2] ../../linbox/algorithms/lattice.inl:351:5: error: 'lllMethod' was not declared in this scope [linbox1.4.2] lllMethod.LLL(); [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:356:5: error: 'wrapper' is not a member of 'fplll' [linbox1.4.2] fplll::wrapper lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:356:20: error: expected ';' before 'lllMethod' [linbox1.4.2] fplll::wrapper lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:358:5: error: 'lllMethod' was not declared in this scope [linbox1.4.2] lllMethod.LLL(); [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:363:5: error: 'heuristic' is not a member of 'fplll' [linbox1.4.2] fplll::heuristic<ZT,double> lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:363:24: error: expected primaryexpression before ',' token [linbox1.4.2] fplll::heuristic<ZT,double> lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:363:25: error: expected primaryexpression before 'double' [linbox1.4.2] fplll::heuristic<ZT,double> lllMethod(&B,meth.getPrecision(), [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:363:25: error: expected ';' before 'double' [linbox1.4.2] ../../linbox/algorithms/lattice.inl:366:5: error: 'lllMethod' was not declared in this scope [linbox1.4.2] lllMethod.LLL(); [linbox1.4.2] ^ [linbox1.4.2] ../../linbox/algorithms/lattice.inl:376:23: error: 'class fplll::ZZ_mat<__mpz_struct [1]>' has no member named 'Get' [linbox1.4.2] H.setEntry(i,j, B.Get(i,j) );
comment:26 followup: ↓ 27 Changed 6 years ago by
If fixing this really requires a lot of work in Linbox, we could just disable fplll in Linbox waiting for a Linbox fix... if that's an option.
comment:27 in reply to: ↑ 26 Changed 6 years ago by
Replying to jpflori:
we could just disable fplll in Linbox waiting for a Linbox fix... if that's an option.
This sounds like the most reasonable choice. I don't think any linbox dependency on fplll is actually exposed to sage at the moment. The hook was kept for the future.
comment:28 Changed 6 years ago by
 Branch changed from u/malb/t21221fplll5 to public/fplll5
 Commit changed from f986fe12472ccd88c79a7691cf2235fe4ca2bb0b to cc8a95f7624eac7c17ac123aebb4ac64e4ef2ab2
 Dependencies set to #17635
 Status changed from needs_review to positive_review
With this strategy everything looks ok. Let the patchbots (or Volker?) do their testing jobs.
Last 10 new commits:
98cc68c  Only try relevant extension for OS when loading libSingular.

399f051  Add cddlib as a dependency and make gfanlib building explicit

797c539  Improve the cleaning

75e383a  use Singular.pc almost everywhere

a0f55ea  Singular.pc is now used everywhere. plural.pyx doesn't need givaro or readline or m anymore.

55447da  Do not build (a second time) libfactory when building Singular.

e5c10ae  Cleaner removal of files.

29077a7  Use wildcards for singular and letterplace in module_list.py.

f366df4  Merge branch 'singular4' into fplll5

cc8a95f  Let Linbox build by disabling its fplll linking.

comment:29 Changed 6 years ago by
Oops, it seems I pushed a wrong branch. I'll do a force push in a few seconds, sorry about that...
comment:30 Changed 6 years ago by
 Commit changed from cc8a95f7624eac7c17ac123aebb4ac64e4ef2ab2 to 3f2d20c8c232770e63442b38e818e7b50133bd85
 Status changed from positive_review to needs_review
comment:31 Changed 6 years ago by
Sorry for the mess, here's a clean branch where I only merged trac/develop
and disabled fplll linking wihtin linbox.
comment:32 Changed 6 years ago by
 Status changed from needs_review to positive_review
comment:33 Changed 6 years ago by
 Reviewers changed from JeanPierre Flori to JeanPierre Flori, Thierry Monteil
 Status changed from positive_review to needs_info
I started the tests before the last changes of JeanPierre, but i got doctests failure on 32bit system, which i guess will still be here after applying the last two commits too.
sage t long warnlong 124.3 src/sage/modules/free_module_integer.py ********************************************************************** File "src/sage/modules/free_module_integer.py", line 78, in sage.modules.free_module_integer.IntegerLattice Failed example: IntegerLattice(A) Expected: Free module of degree 10 and rank 10 over Integer Ring User basis matrix: [ 0 1 2 0 1 2 1 0 1 1] [ 0 1 0 3 0 0 0 0 3 1] [ 1 1 1 0 3 0 0 1 2 2] [1 2 1 1 2 2 1 1 0 1] [ 1 0 4 2 0 1 2 1 0 0] [ 2 3 0 1 1 0 2 3 0 0] [2 3 2 0 0 1 1 1 3 2] [3 0 1 0 2 1 2 1 1 1] [ 1 4 1 1 2 2 1 0 3 1] [1 1 0 3 1 2 2 3 1 0] Got: Free module of degree 10 and rank 10 over Integer Ring User basis matrix: [ 0 1 2 0 1 2 1 0 1 1] [ 1 2 1 1 2 2 1 1 0 1] [ 0 2 2 2 1 2 1 0 1 2] [ 0 1 0 3 0 0 0 0 3 1] [ 1 0 4 2 0 1 2 1 0 0] [ 2 3 0 1 1 0 2 3 0 0] [3 2 3 2 2 1 0 0 0 2] [3 0 1 0 2 1 2 1 1 1] [ 1 4 1 1 2 2 1 0 3 1] [1 1 0 3 1 2 2 3 1 0] ********************************************************************** File "src/sage/modules/free_module_integer.py", line 95, in sage.modules.free_module_integer.IntegerLattice Failed example: sage.crypto.gen_lattice(type='modular', m=10, seed=42, dual=True, lattice=True) Expected: Free module of degree 10 and rank 10 over Integer Ring User basis matrix: [ 0 1 2 0 1 2 1 0 1 1] [ 0 1 0 3 0 0 0 0 3 1] [ 1 1 1 0 3 0 0 1 2 2] [1 2 1 1 2 2 1 1 0 1] [ 1 0 4 2 0 1 2 1 0 0] [ 2 3 0 1 1 0 2 3 0 0] [2 3 2 0 0 1 1 1 3 2] [3 0 1 0 2 1 2 1 1 1] [ 1 4 1 1 2 2 1 0 3 1] [1 1 0 3 1 2 2 3 1 0] Got: Free module of degree 10 and rank 10 over Integer Ring User basis matrix: [ 0 1 2 0 1 2 1 0 1 1] [ 1 2 1 1 2 2 1 1 0 1] [ 0 2 2 2 1 2 1 0 1 2] [ 0 1 0 3 0 0 0 0 3 1] [ 1 0 4 2 0 1 2 1 0 0] [ 2 3 0 1 1 0 2 3 0 0] [3 2 3 2 2 1 0 0 0 2] [3 0 1 0 2 1 2 1 1 1] [ 1 4 1 1 2 2 1 0 3 1] [1 1 0 3 1 2 2 3 1 0] ********************************************************************** File "src/sage/modules/free_module_integer.py", line 203, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer Failed example: L = IntegerLattice(sage.crypto.gen_lattice(type='modular', m=10, seed=42, dual=True)); L Expected: Free module of degree 10 and rank 10 over Integer Ring User basis matrix: [ 0 1 2 0 1 2 1 0 1 1] [ 0 1 0 3 0 0 0 0 3 1] [ 1 1 1 0 3 0 0 1 2 2] [1 2 1 1 2 2 1 1 0 1] [ 1 0 4 2 0 1 2 1 0 0] [ 2 3 0 1 1 0 2 3 0 0] [2 3 2 0 0 1 1 1 3 2] [3 0 1 0 2 1 2 1 1 1] [ 1 4 1 1 2 2 1 0 3 1] [1 1 0 3 1 2 2 3 1 0] Got: Free module of degree 10 and rank 10 over Integer Ring User basis matrix: [ 0 1 2 0 1 2 1 0 1 1] [ 1 2 1 1 2 2 1 1 0 1] [ 0 2 2 2 1 2 1 0 1 2] [ 0 1 0 3 0 0 0 0 3 1] [ 1 0 4 2 0 1 2 1 0 0] [ 2 3 0 1 1 0 2 3 0 0] [3 2 3 2 2 1 0 0 0 2] [3 0 1 0 2 1 2 1 1 1] [ 1 4 1 1 2 2 1 0 3 1] [1 1 0 3 1 2 2 3 1 0] ********************************************************************** File "src/sage/modules/free_module_integer.py", line 513, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ Failed example: L.reduced_basis[0] Expected: (1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 3, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0) Got: (1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 2, 0, 0, 1, 1) **********************************************************************
Concerning src/sage/matrix/matrix_integer_dense.pyx
i have a Timed out (and interrupt failed)
without further explanations, i will run the tests by hand one by one.
I put it in "need info" not "need work" because i have to test the very last commit on a fresh install to confirm the issue, this might take some time since the VM is a bit slow.
comment:34 Changed 6 years ago by
Looks like you're getting a different LLL reduced basis. Mhh, no idea why 32 bit systems might be different, we might have to wait to see what the patchbots report.
comment:35 followup: ↓ 36 Changed 6 years ago by
Will the patchbots do anything if this is not positive_review?
comment:36 in reply to: ↑ 35 Changed 6 years ago by
 Cc jdemeyer added
 Status changed from needs_info to needs_review
On my fresh retry, i got the same failures (but no problem for src/sage/matrix/matrix_integer_dense.pyx
which were probably due to some optional pkgs).
Replying to malb:
Looks like you're getting a different LLL reduced basis. Mhh, no idea why 32 bit systems might be different, we might have to wait to see what the patchbots report.
I do not know either, especially since LLL is deterministic, so perhaps is there some nonconstantlength variable in the floatingpoint part of the code ?
Usually, the arch is sufficient to separate such behaviours (e.g. on my 64bit daily laptop, the tests are OK), and we can make specific tests for them.
But before settling down a #32bit
doctest with those values, we should check that the bases are indeed LLLreduced (it could be an explicit part of the test by the way).
For double checking, note that apart from my patchbot (actually a qemulated Pentium III), arando is the other running 32bit arch, which is run by Jeroen, so we could ask him to tell that patchbot to focus on that ticket.
Replying to jpflori:
Will the patchbots do anything if this is not positive_review?
Yes, since the patchbots are more a tool to help reviewing than to help releasing (for which the buildbot helps). Actually, being in needs_review
gives you more chance to get some patchbots looking at your code (so let me put the ticket to needs_review
, just to motivate some of those): https://github.com/sagemath/sagepatchbot/blob/master/sage_patchbot/patchbot.py#L420 Note that you can force the retest with the ?kick
trick : https://patchbot.sagemath.org/ticket/21221/?kick (this is why there is retry: True
now, i clicked on that link yesterday).
comment:37 Changed 6 years ago by
I can reproduce it in a 32bit virtual machine:
sage: from sage.modules.free_module_integer import IntegerLattice sage: A = sage.crypto.gen_lattice(type='modular', m=10, seed=42, dual=True) sage: copy(A).LLL(algorithm="fpLLL:proved", fp="rr")[3] (1, 2, 1, 1, 2, 2, 1, 1, 0, 1) sage: copy(A).LLL(algorithm="fpLLL:proved", fp="fp")[3] (0, 1, 0, 3, 0, 0, 0, 0, 3, 1)
On a 64bit system:
sage: copy(A).LLL(algorithm="fpLLL:proved", fp="fp")[3] (1, 2, 1, 1, 2, 2, 1, 1, 0, 1) sage: copy(A).LLL(algorithm="fpLLL:proved", fp="rr")[3] (1, 2, 1, 1, 2, 2, 1, 1, 0, 1)
Note that the last outputs differ. In either case the output is LLL reduced (I'm not in favour of adding that test to the doctest though, since LLLreducedness is not the main point there, thus it would be a bit distracting).
A cheap workaround is to simply pick a different seed for the doctest.
comment:38 Changed 6 years ago by
 Commit changed from 3f2d20c8c232770e63442b38e818e7b50133bd85 to c4fa878505265cd2a45a58f182b99e41d0530c64
Branch pushed to git repo; I updated commit sha1. New commits:
c4fa878  unify 32bit and 64bit doctest outputs

comment:39 Changed 6 years ago by
This seems to have done the trick.
comment:40 Changed 6 years ago by
Maybe we should still document that even with the same seed, the LLL reduction, although deterministic, might give different results on 32 and 64 platforms (different roundng for floats?) ?
comment:41 Changed 6 years ago by
 Commit changed from c4fa878505265cd2a45a58f182b99e41d0530c64 to 1a15357fb2e19e014bb313f514aad7528a4c89e3
Branch pushed to git repo; I updated commit sha1. New commits:
1a15357  add warning about different LLL answers

comment:42 Changed 6 years ago by
 Status changed from needs_review to positive_review
Good to go for me.
comment:43 Changed 6 years ago by
Waiting for Volker's run report. But in sageongentoo, I couldn't get the test suite for fpylll
to run successfully when it was built using sagelib
. I decided to hard disable it until I figure if it's broken on my side only or it is for everyone. Actually the fpylll
tarball that is linked in this ticket doesn't include the test suite or does it?
comment:44 Changed 6 years ago by
 Commit changed from 1a15357fb2e19e014bb313f514aad7528a4c89e3 to d194317e800984cdda95a99d4b33d43c88496410
 Status changed from positive_review to needs_review
Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:
d194317  fpylll 0.2.2dev

comment:45 Changed 6 years ago by
 Description modified (diff)
comment:46 Changed 6 years ago by
Pushed fpylll 0.2.2dev which ships the testsuite and doesn't rely on environment variables.
comment:47 followups: ↓ 48 ↓ 50 Changed 6 years ago by
I am not sure that was fair to push that particular upgrade while Volker has already started to test stuff on the bot.
At least the change is minimal and fpylll
doesn't currently have a spkgcheck
nor do we have py.test
in the "vanilla" sage tree to run it. But I do, so I have concerns because it looks to me that fpylll
has had its test suite run without sage with python 2.7 and 3.5 but it looks like its test suite when built against sage is sage testsuite itself. Just for fun
>>> Test phase: devpython/fpylll0.2.1 * python2_7: running distutilsr1_run_phase python_test =============================================================================== test session starts =============================================================================== platform linux2  Python 2.7.10, pytest2.8.7, py1.4.30, pluggy0.3.1  /usr/bin/python2.7 cachedir: .cache rootdir: /scratch2/portage/devpython/fpylll0.2.1/work/fpylll0.2.1dev, inifile: collecting ... collected 17 items / 1 errors tests/test_bkz_python.py::test_bkz_init FAILED tests/test_bkz_python.py::test_simple_bkz_call FAILED tests/test_bkz_python.py::test_bkz_call FAILED tests/test_gso.py::test_gso_init /scratch2/portage/devpython/fpylll0.2.1/temp/environment: line 2590: 5851 Segmentation fault py.test v * python3_4: running distutilsr1_run_phase python_test =============================================================================== test session starts =============================================================================== platform linux  Python 3.4.3, pytest2.8.7, py1.4.30, pluggy0.3.1  /usr/bin/python3.4 cachedir: .cache rootdir: /scratch2/portage/devpython/fpylll0.2.1/work/fpylll0.2.1dev, inifile: collecting ... collected 19 items tests/test_bkz.py::test_bkz_init PASSED tests/test_bkz.py::test_bkz_bkz PASSED tests/test_bkz_python.py::test_bkz_init PASSED tests/test_bkz_python.py::test_simple_bkz_call PASSED tests/test_bkz_python.py::test_bkz_call PASSED tests/test_gso.py::test_gso_init PASSED tests/test_gso.py::test_gso_d PASSED tests/test_gso.py::test_gso_int_gram_enabled PASSED tests/test_gso.py::test_gso_update_gso PASSED tests/test_lll.py::test_lll_init PASSED tests/test_lll.py::test_lll_lll PASSED tests/test_numpy.py::test_dump_mu PASSED tests/test_numpy.py::test_dump_r PASSED tests/test_precision.py::test_precision PASSED tests/test_pruner.py::test_pruner_gso PASSED tests/test_pruner.py::test_pruner_vec PASSED tests/test_random.py::test_randomize PASSED tests/test_simple_bkz.py::test_simple_bkz_init PASSED tests/test_simple_bkz.py::test_simple_bkz_reduction PASSED =========================================================================== 19 passed in 59.30 seconds ============================================================================
python3.4 results to show what happens when not built against sage.
I am not sure how SAGERUNTIME
is generated but fpylll
needs to be in it. From the look of it, it is generated so it is probably OK.
fpylll
needs to be build against sagelib and sagelib needs fpylll
to run. From my packaging point of view that means that
 I will have to split the building of the documentation in its own package, again. Otherwise I have circular dependencies that cannot be satisfied
 I can forget about ever achieving a state where I can take a package for sage and do build, test and install in that order. From now on there is no hope for testing sage before it is fully installed. Looking around it seems very common in python packages, it looks like testing before installing on the same machine is old school.
comment:48 in reply to: ↑ 47 Changed 6 years ago by
Replying to fbissey:
 I will have to split the building of the documentation in its own package, again. Otherwise I have circular dependencies that cannot be satisfied
OK I won't have to split for now. Documentation builds fine without fpylll
installed. Should future documentation make plots that depends directly or indirectly from fpylll
and the split will have to happen.
comment:49 Changed 6 years ago by
 Status changed from needs_review to needs_work
I'm getting some failures on OSX:
sage t long src/sage/matrix/matrix_integer_dense.pyx ********************************************************************** File "src/sage/matrix/matrix_integer_dense.pyx", line 2726, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.? Failed example: A.BKZ() Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.matrix.matrix_integer_dense.Matrix_integer_dense.?[1]>", line 1, in <module> A.BKZ() File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723) BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds), File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192) File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457) SignalError: Segmentation fault ********************************************************************** File "src/sage/matrix/matrix_integer_dense.pyx", line 2732, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.? Failed example: A.BKZ(use_givens=True) Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.matrix.matrix_integer_dense.Matrix_integer_dense.?[3]>", line 1, in <module> A.BKZ(use_givens=True) File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723) BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds), File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192) File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457) SignalError: Segmentation fault ********************************************************************** File "src/sage/matrix/matrix_integer_dense.pyx", line 2738, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.? Failed example: A.BKZ(fp="fp") Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.matrix.matrix_integer_dense.Matrix_integer_dense.?[5]>", line 1, in <module> A.BKZ(fp="fp") File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723) BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds), File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192) File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457) SignalError: Segmentation fault ********************************************************************** 1 item had failures: 3 of 67 in sage.matrix.matrix_integer_dense.Matrix_integer_dense.? [573 tests, 3 failures, 21.83 s]
and
sage t long src/sage/modules/free_module_integer.py ********************************************************************** File "src/sage/modules/free_module_integer.py", line 216, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer Failed example: L.shortest_vector() Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer[2]>", line 1, in <module> L.shortest_vector() File "sage/misc/cachefunc.pyx", line 2038, in sage.misc.cachefunc.CachedMethodCaller.__call__ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:10183) w = self._instance_call(*args, **kwds) File "sage/misc/cachefunc.pyx", line 1914, in sage.misc.cachefunc.CachedMethodCaller._instance_call (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:9647) return self.f(self._instance, *args, **kwds) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/modules/free_module_integer.py", line 633, in shortest_vector w = vector(ZZ, SVP.shortest_vector(L, *args, **kwds)) File "src/fpylll/fplll/svpcvp.pyx", line 71, in fpylll.fplll.svpcvp.shortest_vector (build/src/fpylll/fplll/svpcvp.cpp:3122) File "src/cysignals/signals.pyx", line 102, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1273) RuntimeError: Aborted ********************************************************************** File "src/sage/modules/free_module_integer.py", line 454, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ Failed example: L.BKZ(block_size=10) Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ[6]>", line 1, in <module> L.BKZ(block_size=Integer(10)) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/modules/free_module_integer.py", line 466, in BKZ basis = [v for v in basis.BKZ(*args, **kwds) if v] File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723) BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds), File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192) File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457) SignalError: Segmentation fault ********************************************************************** File "src/sage/modules/free_module_integer.py", line 457, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ Failed example: min(v.norm().n() for v in L.reduced_basis) Expected: 4.12310562561766 Got: 5.19615242270663 ********************************************************************** File "src/sage/modules/free_module_integer.py", line 510, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ Failed example: L.HKZ() Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ[2]>", line 1, in <module> L.HKZ() File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/modules/free_module_integer.py", line 516, in HKZ return self.BKZ(block_size=self.rank()) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/modules/free_module_integer.py", line 466, in BKZ basis = [v for v in basis.BKZ(*args, **kwds) if v] File "sage/matrix/matrix_integer_dense.pyx", line 2856, in sage.matrix.matrix_integer_dense.Matrix_integer_dense.BKZ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/matrix/matrix_integer_dense.cpp:25723) BKZ.reduction(A, BKZ.Param(block_size=block_size, delta=delta, **kwds), File "src/fpylll/fplll/bkz.pyx", line 742, in fpylll.fplll.bkz.bkz_reduction (build/src/fpylll/fplll/bkz.cpp:8192) File "src/cysignals/signals.pyx", line 108, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1457) SignalError: Segmentation fault ********************************************************************** File "src/sage/modules/free_module_integer.py", line 513, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ Failed example: L.reduced_basis[0] Expected: (0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 2) Got: (1, 0, 0, 2, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 2, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0) ********************************************************************** File "src/sage/modules/free_module_integer.py", line 605, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.shortest_vector Failed example: L.shortest_vector().norm().n() Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.shortest_vector[4]>", line 1, in <module> L.shortest_vector().norm().n() File "sage/misc/cachefunc.pyx", line 2038, in sage.misc.cachefunc.CachedMethodCaller.__call__ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:10183) w = self._instance_call(*args, **kwds) File "sage/misc/cachefunc.pyx", line 1914, in sage.misc.cachefunc.CachedMethodCaller._instance_call (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:9647) return self.f(self._instance, *args, **kwds) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/modules/free_module_integer.py", line 633, in shortest_vector w = vector(ZZ, SVP.shortest_vector(L, *args, **kwds)) File "src/fpylll/fplll/svpcvp.pyx", line 71, in fpylll.fplll.svpcvp.shortest_vector (build/src/fpylll/fplll/svpcvp.cpp:3122) File "src/cysignals/signals.pyx", line 102, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1273) RuntimeError: Aborted ********************************************************************** File "src/sage/modules/free_module_integer.py", line 660, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis Failed example: v = L.shortest_vector(update_reduced_basis=False) Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis[4]>", line 1, in <module> v = L.shortest_vector(update_reduced_basis=False) File "sage/misc/cachefunc.pyx", line 2038, in sage.misc.cachefunc.CachedMethodCaller.__call__ (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:10183) w = self._instance_call(*args, **kwds) File "sage/misc/cachefunc.pyx", line 1914, in sage.misc.cachefunc.CachedMethodCaller._instance_call (/Users/buildslavesage/slave/sage_git/build/src/build/cythonized/sage/misc/cachefunc.c:9647) return self.f(self._instance, *args, **kwds) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/modules/free_module_integer.py", line 633, in shortest_vector w = vector(ZZ, SVP.shortest_vector(L, *args, **kwds)) File "src/fpylll/fplll/svpcvp.pyx", line 71, in fpylll.fplll.svpcvp.shortest_vector (build/src/fpylll/fplll/svpcvp.cpp:3122) File "src/cysignals/signals.pyx", line 102, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1273) RuntimeError: Aborted ********************************************************************** File "src/sage/modules/free_module_integer.py", line 661, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis Failed example: L.update_reduced_basis(v) Exception raised: Traceback (most recent call last): File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 498, in _run self.compile_and_execute(example, compiler, test.globs) File "/Users/buildslavesage/slave/sage_git/build/local/lib/python2.7/sitepackages/sage/doctest/forker.py", line 861, in compile_and_execute exec(compiled, globs) File "<doctest sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis[5]>", line 1, in <module> L.update_reduced_basis(v) NameError: name 'v' is not defined ********************************************************************** File "src/sage/modules/free_module_integer.py", line 662, in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis Failed example: bool(L.reduced_basis[0].norm() < B[0].norm()) Expected: True Got: False ********************************************************************** 5 items had failures: 1 of 4 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer 2 of 9 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.BKZ 2 of 5 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.HKZ 1 of 11 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.shortest_vector 3 of 8 in sage.modules.free_module_integer.FreeModule_submodule_with_basis_integer.update_reduced_basis [97 tests, 9 failures, 63.26 s]
comment:50 in reply to: ↑ 47 Changed 6 years ago by
Replying to fbissey:
I am not sure that was fair to push that particular upgrade while Volker has already started to test stuff on the bot.
Sorry, I didn't mean to create more work, rather less.
comment:51 Changed 6 years ago by
Mhh, wo we still have SSH access to an OS X box. It's hard to tell from here what may have caused it. I'll also ask on our fpllldevel mailinglist if people have OSX boxes.
comment:52 Changed 6 years ago by
Looking at the failures, they look like the kind of failures I used to get. Was there maybe a mixup somewhere and an old branch was tested?
comment:53 Changed 6 years ago by
I may try a build now that I finished my clang exploration.
comment:54 followup: ↓ 55 Changed 6 years ago by
Hi there, @fbissey did you end up building this? @vbraun, anything I should do to trigger a retest?
comment:55 in reply to: ↑ 54 Changed 6 years ago by
Replying to malb:
Hi there, @fbissey did you end up building this? @vbraun, anything I should do to trigger a retest?
I did a try twice. Each time I ended up with a mountain of doctests failing for some apparently unrelated reason. I decided to try again from a pristine tree but I haven't got time to do another attempt yet.
comment:56 Changed 6 years ago by
 Status changed from needs_work to positive_review
I can try again ...
comment:57 Changed 6 years ago by
 Status changed from positive_review to needs_work
Its definitely still happening on OSX...
comment:58 Changed 6 years ago by
Thanks & sorry for the noise. I'll dig up an OS X box and debug there.
comment:59 Changed 6 years ago by
This is strange: I built & tested on an OS X box now and all tests passed. Using
 branch: public/fplll5
 fplll: https://github.com/fplll/fplll/releases/download/5.0.3rc0/fplll5.0.3rc0.tar.gz
 fpylll: https://github.com/fplll/fpylll/releases/download/0.2.2dev/fpylll0.2.2dev.tar.gz
Not sure what to do about this now. Any suggestions?
comment:60 Changed 6 years ago by
Apart from getting access to the problematic machine, no...
comment:61 Changed 6 years ago by
On a second OSX box I'm now seeing:
ImportError: dlopen(/Users/vagrant/fplll/lib/python2.7/sitepackages/fpylll/fplll/enumeration.so, 2): Symbol not found: __ZN5fplll14EnumerationDynINS_5FP_NRIA1_10dpe_structEEE16process_solutionEd Referenced from: /Users/vagrant/fplll/lib/python2.7/sitepackages/fpylll/fplll/enumeration.so Expected in: flat namespace in /Users/vagrant/fplll/lib/python2.7/sitepackages/fpylll/fplll/enumeration.so
Any OSX experts out there who can make sense of this? More details at https://github.com/fplll/fplll/issues/205
comment:62 Changed 6 years ago by
It seems this is caused by some bad interaction with brew. Starting from a system without homebrew and installing everything in Sage instead, I don't run into this problem. Running doctests now.
comment:63 Changed 6 years ago by
Doctests pass, so I still cannot reproduce the buildbot failues.
comment:64 Changed 6 years ago by
I'm currently building on the build bot and will try to reproduce.
comment:65 Changed 6 years ago by
So on the OS X buildbot, the current branch is ok as far as the two files mentioned by Volker are cocnerned. I'll now merge with develop to see what changes.
comment:66 Changed 6 years ago by
Hum, just merging trac/develop and issuing make, the tests still pass... Now trying to make distclean and make.
comment:67 Changed 6 years ago by
Ok I can now reproduce it. Maybe openblas related?
comment:68 Changed 6 years ago by
To reproduce, I'd merge trac/develop, make distclean
and make
? I'll give that a try.
comment:69 Changed 6 years ago by
fplll doesn't use any blas, btw, so that shouldn't be it.
comment:70 Changed 6 years ago by
Yes indeed, but I was looking for an easy culprit.
And it has nothing to do with the problem here.
Doing make distclean; configure withblas=atlas; make
leads to the same errors.
comment:71 Changed 6 years ago by
Wait, I don't see the same errors as Volker, at least with the latest build.
It seems fpylll
got built before sagelib
and the interface to sage ismissing.
Strange.
comment:72 Changed 6 years ago by
(And I'm using the correct branch where sagelib
is in the dependencies
file for fpylll
.)
comment:73 Changed 6 years ago by
Forcing the recompilation of fpylll
solved my issues... no more errors...
comment:74 Changed 6 years ago by
Mhhh, sagelib
is not the right thing to put as a dependency
?
comment:75 Changed 6 years ago by
It seems you're right.
Putting sagelib
in dependencies
turns into inst_sagelib
in build/make/Makefile
and this target does not exist.
The creation process of that Makefile should be modified.
Or the field in the dependency file...
comment:76 Changed 6 years ago by
Or put  $(SAGERUNTIME)
in dependencies
.
comment:77 Changed 6 years ago by
The conway_polynomials package depends on sage runtime if you want an example.
comment:78 Changed 6 years ago by
And maybe don't put it as an order only dependency in case fpylll must be recompiled if anything in the integer matrix class changes.
comment:79 Changed 6 years ago by
But SAGERUNTIME
is a phony target so it surely does not matter.
comment:80 Changed 6 years ago by
With a properly built Sage I get no segfault. So maybe Volker setup ended up in broken state.
comment:81 Changed 6 years ago by
 Commit changed from d194317e800984cdda95a99d4b33d43c88496410 to 848f2119c022070058e9fbae813d8fbd0a0a4c03
Branch pushed to git repo; I updated commit sha1. New commits:
848f211  add dependency on SAGERUNTIME

comment:82 Changed 6 years ago by
 Status changed from needs_work to positive_review
comment:83 Changed 6 years ago by
 Branch changed from public/fplll5 to 848f2119c022070058e9fbae813d8fbd0a0a4c03
 Resolution set to fixed
 Status changed from positive_review to closed
comment:84 Changed 6 years ago by
 Commit 848f2119c022070058e9fbae813d8fbd0a0a4c03 deleted
Breakage: #21675
New commits:
fplll 5.0.1 & drop sage.libs.fplll
add fpylll
A.LLL() and A.BKZ() using fpylll