Upgrade to gfan 0.6.2

[rws]

gfan 0.6.2 standalone builds nearly out of the box with clang-check, it is compiled with -std=c++0x, meaning no adaptations are necessary for C++11.

• ​http://home.math.au.dk/jensen/software/gfan/gfan0.6.2.tar.gz

We also add spkg-check and fix a test there.

[rws]

This compiles cleanly with std=c++11 but fails at least sage -t --warn-long 42.1 src/sage/rings/polynomial/groebner_fan.py # 9 doctests failed

---

New commits:

​56e23eb	23353: chksum/pkg version
​f697b57	23353: fix spkg-install, Makefile-patch

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​cb88983

23353: doctest fixes

[fbissey]

I think we should push more of our makefile changes "upstream". In a more diplomatic fashion if we want it to be picked up by the "real" upstream. But I definitely think we should make our makefile patch unnecessary.

[fbissey]

In light of what is happening with lcalc in #23341, do CXXFLAGS from spkg-install get used during the compilation?

[fbissey]

​https://github.com/rwst/gfan06/pull/1 is ready for review for the makefile. I could merge it directly but prefer to submit it to third party scrutiny first.

[rws]

Replying to fbissey:

In light of what is happening with lcalc in #23341, do CXXFLAGS from spkg-install get used during the compilation?

Yes, after I fixed the previous (0.5) Makefile patch, it couldn't work before because CXXFLAGS_LOCAL was set but not used. Note I'm using export in the spkg-install. I couldn't get other ways to work.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​25811d6

23353: improved Makefile

[rws]

The first error is this:

sage: R.<x,y,z> = PolynomialRing(QQ,3)
sage: f1 = x*y*z - 1
sage: f2 = f1*(x^2 + y^2 + z^2)
sage: f3 = f2*(x + y + z - 1)
sage: I = R.ideal([f1,f2,f3])
sage: gf = I.groebner_fan()
sage: pf = gf.tropical_intersection(symmetry_generators = '(1,2,0),(1,0,2)')

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-890375cd29eb> in <module>()
      5 I = R.ideal([f1,f2,f3])
      6 gf = I.groebner_fan()
----> 7 pf = gf.tropical_intersection(symmetry_generators = '(1,2,0),(1,0,2)')

/home/ralf/sage/local/lib/python2.7/site-packages/sage/rings/polynomial/groebner_fan.pyc in tropical_intersection(self, parameters, symmetry_generators, *args, **kwds)
   1713                 cmd = cmd + ' --symmetryExploit'
   1714                 id_str = id_str + '{' + symmetry_generators + '}'
-> 1715             f = self.gfan(cmd=cmd, I = id_str)
   1716             pf = TropicalPrevariety(f,self.ideal().gens(), self.ring(), parameters = parameters)
   1717             pf._gfan_output = f

/home/ralf/sage/local/lib/python2.7/site-packages/sage/rings/polynomial/groebner_fan.pyc in gfan(self, cmd, I, format)
   1115         cmd += self._gfan_mod()
   1116         print("Error running gfan command %s on %s"%(cmd, self))
-> 1117         s = gfan(I, cmd, verbose=self.__verbose, format=format)
   1118         if s.strip() == '{':
   1119             raise RuntimeError("Error running gfan command %s on %s"%(cmd, self))

/home/ralf/sage/local/lib/python2.7/site-packages/sage/interfaces/gfan.pyc in __call__(self, I, cmd, verbose, format)
     68         # we avoid interpreting this as an error
     69         if (len(err) > 0) and not (err.startswith('LP algorithm being used:')):
---> 70             raise RuntimeError(err)
     71 
     72         return ans

RuntimeError: _application PolyhedralCone
_version 2.2
_type PolyhedralCone

AMBIENT_DIM
3

DIM
3

IMPLIED_EQUATIONS

LINEALITY_DIM
1

LINEALITY_SPACE
1 1 1

FACETS
0 1 -1
1 -1 0

RELATIVE_INTERIOR_POINT
2 1 0

Error running gfan command tropicalintersection --symmetryExploit on Groebner fan of the ideal:
Ideal (x*y*z - 1, x^3*y*z + x*y^3*z + x*y*z^3 - x^2 - y^2 - z^2, x^4*y*z + x^3*y^2*z + x^2*y^3*z + x*y^4*z + x^3*y*z^2 + x*y^3*z^2 + x^2*y*z^3 + x*y^2*z^3 + x*y*z^4 - x^3*y*z - x*y^3*z - x*y*z^3 - x^3 - x^2*y - x*y^2 - y^3 - x^2*z - y^2*z - x*z^2 - y*z^2 - z^3 + x^2 + y^2 + z^2) of Multivariate Polynomial Ring in x, y, z over Rational Field

The last paragraph is output from a print command I inserted.

The second fail in rings follows from this.

[fbissey]

Replying to rws:

Replying to fbissey:

In light of what is happening with lcalc in #23341, do CXXFLAGS from spkg-install get used during the compilation?

Yes, after I fixed the previous (0.5) Makefile patch, it couldn't work before because CXXFLAGS_LOCAL was set but not used. Note I'm using export in the spkg-install. I couldn't get other ways to work.

All right, so you really see -std=c++11 in your logs then? I couldn't ever get it to work in #23341 - still thinking about that.

[rws]

Replying to fbissey:

All right, so you really see -std=c++11 in your logs then?

Yes.

[vbraun]

There is now a gfan-0.6.2 at ​http://home.math.au.dk/jensen/software/gfan/gfan0.6.2.tar.gz

[vbraun]

Every time I look at the gfan source code I'm having bad feelings about this...

---

New commits:

​02612ef

Update to gfan 0.6.2 and fix trivial output changes

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2931494

Remove the wonky -fno-guess-branch-probability compiler flag

[vbraun]

The makefile says

# When compiling symmetrictraversal.cpp as any other file, it causes the program to crash in certain situations.
# (compiling with gcc version 4.7.2 and running gfan _tropicaltraverse on a starting cone for Grassmann3_7)
# Either this is a bug in the code or in the compiler. The bug disappears by compiling with -fno-guess-branch-probability

but clang doesn't support that, and gcc 4 is ancient...

[fbissey]

Are you thinking this will help with the xcode issue?

[vbraun]

Giving it a trial run right now

[vbraun]

Still doesn't work... we should just revert clang and use gcc until its fixed

[fbissey]

Replying to vbraun:

Still doesn't work... we should just revert clang and use gcc until its fixed

That's a real pity, but I have to agree. Until we either find that we screwed up somewhere or that clang is the broken thing we'll have to stay with gcc if we want a release now.

[dimpase]

I wrote on sage-dev that cython fails its own test in c++ extension on the latest xcode. (zero reaction there, so here it goes:)

[cython-0.28.1] ERROR: runTest (__main__.CythonRunTestCase)
[cython-0.28.1] compiling (cpp) and running libcpp_algo
[cython-0.28.1] ----------------------------------------------------------------------
[cython-0.28.1] Traceback (most recent call last):
[cython-0.28.1]   File "runtests.py", line 1114, in run
[cython-0.28.1]     ext_so_path = self.runCompileTest()
[cython-0.28.1]   File "runtests.py", line 803, in runCompileTest
[cython-0.28.1]     self.test_directory, self.expect_errors, self.expect_warnings, self.annotate)
[cython-0.28.1]   File "runtests.py", line 1050, in compile
[cython-0.28.1]     so_path = self.run_distutils(test_directory, module, workdir, incdir)
[cython-0.28.1]   File "runtests.py", line 973, in run_distutils
[cython-0.28.1]     build_extension.run()
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/distutils/command/build_ext.py", line 340, in run
[cython-0.28.1]     self.build_extensions()
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/distutils/command/build_ext.py", line 449, in build_extensions
[cython-0.28.1]     self.build_extension(ext)
[cython-0.28.1]   File "runtests.py", line 486, in build_extension
[cython-0.28.1]     _build_ext.build_extension(self, ext)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
[cython-0.28.1]     depends=ext.depends)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/site-packages/numpy/distutils/ccompiler.py", line 343, in CCompiler_compile
[cython-0.28.1]     single_compile(o)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/site-packages/numpy/distutils/ccompiler.py", line 303, in single_compile
[cython-0.28.1]     self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
[cython-0.28.1]   File "/Users/dimpase/Sage/sagetrac-mirror/local/lib/python2.7/site-packages/numpy/distutils/unixccompiler.py", line 61, in UnixCCompiler__compile
[cython-0.28.1]     raise CompileError(msg)
[cython-0.28.1] CompileError: Command "cc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wno-unused -I/Users/dimpase/Sage/sagetrac-mirror/local/var/tmp/sage/build/cython-0.28.1/src/tests/run -I/Users/dimpase/Sage/sagetrac-mirror/local/include/python2.7 -c libcpp_algo.cpp -o /Users/dimpase/Sage/sagetrac-mirror/local/var/tmp/sage/build/cython-0.28.1/src/TEST_TMP/run/cpp/libcpp_algo/libcpp_algo.o -DCYTHON_REFNANNY=1" failed with exit status 1
[cython-0.28.1]
[cython-0.28.1] ----------------------------------------------------------------------
[cython-0.28.1] Ran 10737 tests in 3506.735s

Perhaps it is related?

[rws]

Another data point, since rc0 or rc1 I get a SIGFPE when starting sage -gdb. I should hunt the exact commit before opening the ticket.

[vbraun]

This is probably #25092

[dimpase]

See also #25118: gfan fails when compiled with XCode 9.3 on OSX (depending upon CPU, it's OK on Xeons, but is broken on i5/i7)

[dimpase]

How about adding spkg-check ? There is make check way to run tests.

By the way, check out #25118 - there is a fix for the OSX problem, I'm just not 100% sure it's totally safe to just drop that delete statement.

[dimpase]

this adds spkg-check and the necessary change to the source to fail on failing tests. Also, the port of #25118 patch for Xcode 6.3.

One problem to sort out: the test 0008PolynomialSetUnion exhibits a sort of bug, due to output being unsorted, despite sort called on it. Does sort have another meaning while spkg-check is called?

[dimpase]

The testing bug is most probably in src/app_test.cpp, invoked via ./gfan _test from the Makefile's check target, which generates these lines such as

...
Running command:"cat <testsuite/0008PolynomialSetUnion/input|./gfan _bases | ./gfan _polynomialsetunion |sort>testsuite/0008Polynomia
lSetUnion/outputNew"
Running command:"cat <testsuite/0009RenderStairCase/input|./gfan _bases --symmetry | ./gfan _renderstaircase -m -w6 -d16>testsuite/00
09RenderStairCase/outputNew"
...

and parses the output somehow (yes, in C++ !). I checked the intermediate results of this long chain of | for 0008PolynomialSetUnion, they look OK.

[dimpase]

The testing bug is pretty weird. Basically, there is a C++ test driver, source in src/app_test.cpp, invoked via ./gfan _test, which works on directories (say, bla) in testsuite/. Each directory contains text files input, output, outputNew, and commaand, and does the following:

1. reads the template from command in testsuite/bla/, which looks like %s _bases | %s _polynomialsetunion |sort,

2. replaces %s with ./gfan, prepends the input by prepending cat <testsuite/bla/input|, and output by appending >testsuite/outputNew.

3. executes the resulting chain of shell commands, like cat <testsuite/bla/input|./gfan _bases | ./gfan _polynomialsetunion |sort>testsuite/bla/outputNew, via system()

4. compares output with outputNew for equality; if they are equal, test is successful, otherwise not.

---

Now, one of the tests fails, due to sort producing a weirdly ordered file: while the output should look like

{
a...
...
b...
...
Q[a,b,c]

one gets the 1st line swapped with the last line, i.e.

Q[a,b,c]
a...
...
b...
...
{

Not all the tests call sort, the failing one happens to be the 1st one calling sort.

Worst of all, a repeat of the run of the testsuite produces no error, e.g. if I go to the working directory of the package build and run make check again, everything works.

And this does not limit itself to one OS, I see this on Linux (Gentoo and Ubuntu 16.04) and on OSX.

---

I really don't know how to fix this properly, or even what possibly can cause this, other than a subtle bash (or whatever shell system() uses) bug or something like this. Possible ways out I see

1. calling testsuite twice in row (trying this in spkg-check now)

2. replacing ./gfan _test with a script; the input lines are already generated (sans a call to diff, which should not be hard to add)

3. remove this test

4. ???

[vbraun]

Locale influences sort order, try setting LC_ALL=C

[dimpase]

Replying to vbraun:

Locale influences sort order, try setting LC_ALL=C

I tried replacing |sort>... with | LC_ALL=C sort>..., to no avail. Should I instead export LC_ALL in spkg-check ?

[fbissey]

Would sort -f (ignore case) produce a more stable output?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dd61918

sort output of test 0008 with LC_ALL=C, remove 1st and last lines

[dimpase]

This did cost me half a day... :/ - needs review, anyway!

[dimpase]

removed the link to ​https://github.com/rwst/gfan06/releases/download/0.6.p0/gfan-0.6.tar.bz2 from the ticket description, as we now use the upstream 0.6.2 (fixed this link too), and the corresponding comment:

Note also that there are two versions of the package gfanlib (0.6 and 0.6.2) with changes in 0.6.2 that could be applied equivalently to files in gfan0.6. However, the file names are different!---the prefix gfanlib_ is replaced by app_.

[dimpase]

On Ubuntu xenial, i.e. 16.04, I get

[gfan-0.6.2.p0] g++  -std=c++11 -DNOCDDPREFIX -DGMPRATIONAL -Wuninitialized -fno-omit-frame-pointer -O2   -g     -lpthread -o gfan -L/home/dima/sagetrac-mirror/local/lib -Wl,-rpath,/home/dima/sagetrac-mirror/local/lib   src/polynomialgcd.o src/lp_cdd.o src/parser.o src/field.o src/monomial.o src/printer.o src/polynomial.o src/termorder.o src/term.o src/vektor.o src/division.o src/buchberger.o src/wallideal.o src/lp.o src/enumeration.o src/ep_standard.o src/ep_xfig.o src/reversesearch.o src/application.o src/timer.o src/renderer.o src/field_rationals.o src/symmetry.o src/breadthfirstsearch.o src/genericwalk.o src/minkowskisum.o src/newtonpolytope.o src/tropical.o src/dimension.o src/bergman.o src/subspace.o src/polyhedralcone.o src/gfanapplication.o src/polyhedralfan.o src/tropical2.o src/field_zmodpz.o src/binomial.o src/matrix.o src/latticeideal.o src/scarf.o src/xfig.o src/halfopencone.o src/lll.o src/multiplicity.o src/substitute.o src/polymakefile.o src/saturation.o src/determinant.o src/polynomialring.o src/log.o src/tropicalbasis.o src/symmetriccomplex.o src/linalg.o src/minors.o src/continuedfractions.o src/triangulation.o src/minkowskidual.o src/regularsubdivision.o src/fieldlp.o src/field_rationalfunctions.o src/tropical_weildivisor.o src/intsinpolytope.o src/lattice.o src/graph.o src/restrictedautoreduction.o src/tropicaltraverse.o src/groebnerengine.o src/ge_gfan.o src/nbody.o src/codimoneconnectedness.o src/tropicalmap.o src/traverser_tropical.o src/traverser_groebnerfan.o src/field_rationalfunctions2.o src/mixedvolume.o src/traverser_stableintersection.o src/traverser_secondaryfan.o src/linalgfloat.o src/primarydecomposition.o src/tropicaldeterminant.o src/determinantpoly.o src/traverser_sphere.o src/gfanlib_zcone.o src/gfanlib_symmetry.o src/gfanlib_symmetriccomplex.o src/gfanlib_polyhedralfan.o src/gfanlib_zfan.o src/gfanlib_polymakefile.o src/gfanlib_mixedvolume.o src/gfanlib_circuittableint.o src/gfanlib_paralleltraverser.o src/padic.o src/integergb.o src/traverser_resultantfan.o src/bsptree.o src/traverser_resultantfanspecialization.o src/myassert.o src/traverser_bsptree.o src/gfanlib_traversal.o src/tropicalcurve.o src/packedmonomial.o src/gmpallocator.o src/app_main.o src/app_buchberger.o src/app_doesidealcontain.o src/app_facets.o src/app_groebnercone.o src/app_homogeneityspace.o src/app_homogenize.o src/app_initialforms.o src/app_interactive.o src/app_isgroebnerbasis.o src/app_ismarkedgroebnerbasis.o src/app_krulldimension.o src/app_leadingterms.o src/app_multiplymatrix.o src/app_polynomialsetunion.o src/app_render.o src/app_renderstaircase.o src/app_stats.o src/app_substitute.o src/app_supportindices.o src/app_tolatex.o src/app_transposematrix.o src/app_tropicalbasis.o src/app_tropicalintersection.o src/app_tropicalstartingcone.o src/app_tropicaltraverse.o src/app_walk.o src/app_weightvector.o src/app_scarfisgeneric.o src/app_scarfvisualize.o src/app_scarfcomplex.o src/app_sturmsequence.o src/app_latticeideal.o src/app_lll.o src/app_tropicalmultiplicity.o src/app_idealintersection.o src/app_test.o src/app_saturation.o src/app_idealproduct.o src/app_representatives.o src/app_tropicallifting.o src/app_topolyhedralfan.o src/app_tropicalbruteforce.o src/app_secondaryfan.o src/app_composepermutations.o src/app_minors.o src/app_tropicalrank.o src/app_minkowski.o src/app_triangulate.o src/app_tropicallinearspace.o src/app_combinerays.o src/app_regularsubdivision.o src/app_lpsolve.o src/app_tropicalweildivisor.o src/app_lattice.o src/app_intsinpolytope.o src/app_tropicalevaluation.o src/app_smalessixth.o src/app_smalessixth2.o src/app_nbody.o src/app_spolynomial.o src/app_link.o src/app_normalfancleanup.o src/app_tropicalfunction.o src/app_volume.o src/app_isconnected.o src/app_tropicalhypersurface.o src/app_product.o src/app_commonrefinement.o src/app_tropicalimage.o src/app_groebnerfan.o src/app_fanhomology.o src/app_genericlinearchange.o src/app_mixedvolume.o src/app_fiberpolytope.o src/app_symmetries.o src/app_evaluate.o src/app_exponentlattice.o src/app_minimalassociatedprimes.o src/app_realroots.o src/app_initialdeterminant.o src/app_fansubfan.o src/app_fancones.o src/app_issmooth.o src/app_fancoarsening.o src/app_pointconfiguration.o src/app_librarytest.o src/app_padic.o src/app_integergb.o src/app_matrixproduct.o src/app_traversetropicalintersection.o src/app_markpolynomialset.o src/app_tropicalhypersurfacereconstruction.o src/app_resultantfan.o src/app_isbalanced.o src/app_polytopealgebra.o src/app_debug.o src/app_randompolynomials.o src/app_tropicalcurve.o src/app_tropicalhomotopy.o src/app_integerfactorization.o src/app_tropicalvarietyspan.o src/app_chowbetti.o src/symmetrictraversal.o -lcddgmp -lgmp -lm   -g
[gfan-0.6.2.p0] src/gfanlib_paralleltraverser.o: In function `std::thread::thread<void (&)(gfan::ThreadContext*), gfan::ThreadContext*&>(void (&)(gfan::ThreadContext*), gfan::ThreadContext*&)':
[gfan-0.6.2.p0] /usr/include/c++/5/thread:137: undefined reference to `pthread_create'

Any idea what's wrong there?

This is with gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9).

Is it something to do with -lpthread vs -pthread ? It's the 1st time I see this, advice welcome.

[dimpase]

Indeed, this 1-character change (lpthread to pthread)

diff --git a/build/pkgs/gfan/patches/Makefile.patch b/build/pkgs/gfan/patches/Makefile.patch
index 127e2b2..3e2866b 100644
--- a/build/pkgs/gfan/patches/Makefile.patch
+++ b/build/pkgs/gfan/patches/Makefile.patch
@@ -125,7 +125,7 @@ index 9556cf4..40127b6 100644
  $(MAIN): $(OBJECTS)
  #     $(CCLINKER) $(OBJECTS) $(ADDITIONALLINKOPTIONS) $(GPROFFLAG) -lpthread  -o $(MAIN)
 -      $(CCLINKER) $(OBJECTS) $(ADDITIONALLINKOPTIONS) $(GPROFFLAG) -lpthread -rdynamic -o $(MAIN)
-+      $(CXX) $(CXXFLAGS) -lpthread -o $(MAIN) $(LDFLAGS) $(OBJECTS) $(ADDITIONALLINKOPTIONS) $(GPROFFLAG)
++      $(CXX) $(CXXFLAGS) -pthread -o $(MAIN) $(LDFLAGS) $(OBJECTS) $(ADDITIONALLINKOPTIONS) $(GPROFFLAG)
  
  release:
        rm -f -r $(RELEASEDIR)/*

makes it build.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​420215f

fix pthread vs lpthread issue

[dimpase]

OK, this works now on Linux ad OSX

[fbissey]

So -fno-guess-branch-probability was not needed for you with gcc. That's good. I think this tick all the boxes on what we have talked in this ticket and on sage-devel. So I am happy to send it to the bots.