Upgrade Normaliz version to 3.6.3 and PyNormaliz to 1.19

[mkoeppe]

Upstream:

​Normaliz:

(cd upstream && wget 'https://github.com/Normaliz/Normaliz/releases/download/v3.6.3/normaliz-3.6.3.tar.gz')

​PyNormaliz 1.19:

(cd upstream && wget 'https://files.pythonhosted.org/packages/7f/e9/3c15d1fc8ebba8a2ed17fc0f209804dc1240166a686d2f592305efc9d25e/PyNormaliz-1.19.tar.gz') 

Some more helper functions were included to facilitate the interfacing.

[mkoeppe]

Last 10 new commits:

​737552d	Updating patch with upstream fix for wrong number of lattice points.
​530302d	Merge branch 'u/tscrim/upgrade_noramliz_pynormaliz-22984' of git://trac.sagemath.org/sage into u/tscrim/upgrade_noramliz_pynormaliz-22984
​57d40b0	Upgrade Normaliz to 3.5.2.
​6ba442d	Adding tests from comment:24,25 of #22984.
​183087a	Upgrade PyNormaliz to 1.12
​7f47ec9	Adapted the polyhedron docstring
​117e428	Upgrade normaliz to 3.5.3
​7057ef4	Merge branch to get docstring adaptation
​b6ae686	Merge branch 'develop' of git://trac.sagemath.org/sage into t/25090/pynormaliz2
​eaedcc2	Update PyNormaliz to 1.14

[jipilab]

New commits:

​4dbdc7d

Updated the Cone calls format

[jipilab]

New commits:

​1dcef4c	_init_from_normaliz_data: New, use it in _init_from_*representation
​fcd8101	Updated pynormaliz to 1.15

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​19d8539

Polyhedron_normaliz: In verbose mode, write out normaliz format files

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​4cfcd14

_normaliz_format: Fix last change

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​da8ae7a

Added data and cone methods to normaliz backend

[jipilab]

After a discussion with gh-sebasguts: A new version of pynormaliz will be released in the upcoming days, including extended documentation and latest changes to pynormaliz.

In order not to mix things up too much with #25097 and #25091, it should be put as a priority in the reviewing queue...

[jipilab]

The version numbers are now fixed.

[jipilab]

I merged the changes done over the week with a fresh branch on top of sage8.2.rc2.

To be tested!

---

New commits:

​aa0e2ca	Merge branch normaliz3.5.3 and pynormaliz1.12 into sage8.2.rc2
​2bbd8a6	Update normaliz and pynormaliz
​82d7190	Merged old changes with latest versions

[jipilab]

Both compile without issues on debian-stretch.

[jipilab]

I am current in Osnabrück with Winfried, and perhaps now would be a good time to refresh this ticket, now that the version 1.12 and other update of Normaliz are already in the box...

There will be a new version of Normaliz released in the next few days, so I would wait until the upstream is updated and then change the ticket. I would do the same with pynormaliz and then go ahead to test it.

Further, with that release, it will make it easier to work with the algebraic polytopes ticket.

[jipilab]

New commits:

​5f0535c	Merge branch update_normaliz to sage8.3.rc0
​476bbfa	Updated normaliz and pynormaliz versions

[jipilab]

I guess now this ticket needs proper review and testing.

It is a really good corner stone for the next steps in #25091 and #25097.

Please install on your platforms and check if it compiles...

There are a few new features along with the new versions.

[jipilab]

I get a segmentation fault while testing 'backend_normaliz.py' while running

sage: p = Polyhedron(backend='normaliz')
sage: TestSuite(p).run(skip="_test_pickling")

on the empty polyhedron.

[jipilab]

The traceback I get is pasted below. Something wrong in PyNormaliz? with "._test_nonzero_equal()" ?

sage: p = Polyhedron(backend='normaliz')
sage: TestSuite(p).run(skip="_test_pickling",verbose=True)
running ._test_category() . . . pass
running ._test_eq() . . . pass
running ._test_new() . . . pass
running ._test_nonzero_equal() . . .------------------------------------------------------------------------
/home/jplabbe/sage/local/lib/python2.7/site-packages/cysignals/signals.so(+0x6ad5)[0x7f4ba65eaad5]
/home/jplabbe/sage/local/lib/python2.7/site-packages/cysignals/signals.so(+0x6b7f)[0x7f4ba65eab7f]
/home/jplabbe/sage/local/lib/python2.7/site-packages/cysignals/signals.so(+0xaf84)[0x7f4ba65eef84]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f4bafb040c0]
/home/jplabbe/sage/local/lib/libnormaliz.so.3(_ZN11libnormaliz6MatrixI10__gmp_exprIA1_12__mpz_structS3_EEC2Emm+0x9)[0x7f48e2c7ba29]
/home/jplabbe/sage/local/lib/libnormaliz.so.3(_ZN11libnormaliz4ConeI10__gmp_exprIA1_12__mpz_structS3_EE25prepare_input_constraintsERKSt3mapINS_4Type9InputTypeESt6vectorIS9_IS4_SaIS4_EESaISB_EESt4lessIS8_ESaISt4pairIKS8_SD_EEE+0x37)[0x7f48e2cd83e7]
/home/jplabbe/sage/local/lib/libnormaliz.so.3(_ZN11libnormaliz4ConeI10__gmp_exprIA1_12__mpz_structS3_EE25process_multi_input_innerERSt3mapINS_4Type9InputTypeESt6vectorIS9_IS4_SaIS4_EESaISB_EESt4lessIS8_ESaISt4pairIKS8_SD_EEE+0x8c5)[0x7f48e2d1ef75]
/home/jplabbe/sage/local/lib/libnormaliz.so.3(_ZN11libnormaliz4ConeI10__gmp_exprIA1_12__mpz_structS3_EE19process_multi_inputERKSt3mapINS_4Type9InputTypeESt6vectorIS9_IS4_SaIS4_EESaISB_EESt4lessIS8_ESaISt4pairIKS8_SD_EEE+0xae)[0x7f48e2d2151e]

[gh-sebasguts]

Interesting. I will have a look.

[Winfried]

Please let me know if you think that the problem is in Normaliz and send me the input.

[jipilab]

Replying to gh-sebasguts:

Interesting. I will have a look.

Any idea where the segfault appears?

[Winfried]

It would be good to see the definition of the empty polytope. Then we can investigate the issue further.

[Winfried]

The empty polyhedron runs into a Normaliz segmentation fault if the volume is computed. Will be taken care of.

[jipilab]

I made a verbose=True in the TestSuite to see where it fails, and in the _test_nonzero_equal() it calls bool(self) and self.parent().zero() which both produce segfaults on their own:

sage: P_empty = Polyhedron()
sage: bool(P_empty)
True
sage: p = Polyhedron(vertices=[(1, 1)], rays=[(0, 1)],backend='normaliz',verbose=True)
# Calling PyNormaliz.NmzCone(**{'subspace': [], 'vertices': [[1, 1, 1]], 'cone': [[0, 1]]})
# ----8<---- Equivalent Normaliz input file ----8<----
amb_space 2
subspace 0
vertices 1
 1 1 1
cone 1
 0 1
# ----8<-------------------8<-------------------8<----
sage: bool(p)
True
sage: p_empty = Polyhedron(backend='normaliz',verbose=True)
sage: bool(p_empty)
------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred.
This probably occurred because a *compiled* module has a bug
in it and is not properly wrapped with sig_on(), sig_off().
Python will now terminate.
------------------------------------------------------------------------
/usr/local/bin/sage: line 3:  5585 Segmentation fault      ~/sage/./sage $@


sage: p_empty = Polyhedron(backend='normaliz',verbose=True)
sage: p_empty.parent()
Polyhedra in ZZ^0
sage: p_empty.parent().zero()
------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred.
This probably occurred because a *compiled* module has a bug
in it and is not properly wrapped with sig_on(), sig_off().
Python will now terminate.
------------------------------------------------------------------------
/usr/local/bin/sage: line 3:  6354 Segmentation fault      ~/sage/./sage $@

Note that to create an empty polyhedron with the backend normaliz, no call to normaliz is done. So I do not think that the problem is at this level, but perhaps when dealing with the parents, it tries to call normaliz with something empty and it does not work...

[jipilab]

I could figure out where the segfault occurs and with which input:

# Calling PyNormaliz.NmzCone(**{'inhom_equations': [], 'inhom_inequalities': [[0]]})
# ----8<---- Equivalent Normaliz input file ----8<----
amb_space 0
inhom_equations 0
inhom_inequalities 1
 0
# ----8<-------------------8<-------------------8<----

The problem occurs in the _init_from_Hrepresentation, where we get the normaliz data: {'inhom_equations': [], 'inhom_inequalities': [[0]]}

Somehow normaliz/pynormaliz does not like that?

I believe that dealing with the empty polyhedron at the normaliz level might help here?

[Winfried]

The segmentation fault with Volume is not the cause of this problem. It cannot be reproduced with Normaliz itself because amb_space 0 is not a legal value so that Normaliz terminates with an error message.

[Winfried]

amb_space 0
inhom_equations 0
inhom_inequalities 1
 0

does NOT define the empty polytope. The origin in the 0-dimensional space satisfies the inequality. If you replace the last0 by -1, you obtain the empty polytope.

[jipilab]

Replying to Winfried:

amb_space 0
inhom_equations 0
inhom_inequalities 1
 0

does NOT define the empty polytope. The origin in the 0-dimensional space satisfies the inequality. If you replace the last0 by -1, you obtain the empty polytope.

Oh, interesting, so probably this would be a good way to initialize the empty polyhedron from the H-representation to pass it to normaliz.

I'll check if I can fix that.

[mkoeppe]

The current backend_normaliz handles the empty polyhedron specially (actually sage has one empty polyhedron for each ambient space dimension!) If this special casing could be removed completely with the current version of normaliz, that would be great!

[jipilab]

Replying to mkoeppe:

The current backend_normaliz handles the empty polyhedron specially (actually sage has one empty polyhedron for each ambient space dimension!) If this special casing could be removed completely with the current version of normaliz, that would be great!

Agreed! I'll have a look if this would work with the new version.

[Winfried]

The embedding dimension is an essential parameter in Normaliz. If any simplification is desired, it must be done on the Sage side.

Sebastian and I have tried to correct all problems related to empty polytopes and to embedding dimension 0. Please pull normaliz/master and test again.

[mkoeppe]

#25707 added a cocoalib package to sage. Should consider adding it as a dependency; but it's currently an experimental package

[jipilab]

How to properly get the 0-dimensional polytope consisting of the origin only in the 0-dimensional ambient space using the V-representation input consisting of subspace, vertices and cone in normaliz?

# Calling PyNormaliz.NmzCone(**{'subspace': [], 'vertices': [[1]], 'cone': []})
# ----8<---- Equivalent Normaliz input file ----8<----
amb_space 0
subspace 0
vertices 1
 1
cone 0
# ----8<-------------------8<-------------------8<----

Somehow normaliz does not like the above, which is the format that sage currently gives to normaliz to create the .zero() element of the parent.

[Winfried]

I have run Normaliz (present master branch on GitHub) on this input. No problem in the output:

1 lattice points in polytope (module generators)
0 Hilbert basis elements of recession monoid
1 vertices of polyhedron
0 extreme rays of recession cone
1 support hyperplanes of polyhedron (homogenized)

embedding dimension = 1
affine dimension of the polyhedron = 0 (maximal)
rank of recession monoid = 0 (polyhedron is polytope)
internal index = 1

dehomogenization:
1 


module rank = 1

***********************************************************************

1 lattice points in polytope (module generators):
 1

0 Hilbert basis elements of recession monoid:

1 vertices of polyhedron:
 1

0 extreme rays of recession cone:

1 support hyperplanes of polyhedron (homogenized):
 1

I think this is correct.The vector [1] is the origin in the 0-dim affine space after homogenization.

[jipilab]

Ok, I do not have the latest "normaliz" on my computer...

I do not know exactly how to make sage use a different normaliz than the one installed _within_ sage when doing the usual sage -i normaliz.

Anyhow, if it is possible to get the empty polyhedron and the 0-dim polytope in the latest version of normaliz. Then these problems should be fixed. Would it be reasonable to have a tiny release of normaliz soon and we could update this branch?

[Winfried]

I think the treatment of degenerate situations is o.k. now in normaliz. These minutes I am testing them on Qnormaliz. Once it is o.k., I will malke release 3.6.2.

[jipilab]

Replying to Winfried:

I think the treatment of degenerate situations is o.k. now in normaliz. These minutes I am testing them on Qnormaliz. Once it is o.k., I will malke release 3.6.2.

Great! Thanks a lot!

[Winfried]

Normaliz 3.6.2 released

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​eeab39f	Merge tag '8.4.beta2' into t/25090/public/pynormaliz1.17_normaliz3.6.1
​976b93f	Update normaliz to 3.6.3, pynormaliz to 1.19

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e36973a

build/pkgs/normaliz/spkg-install: Update comment

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​98d5bd3

Merge remote-tracking branch 'trac/develop' into t/25090/public/pynormaliz1.17_normaliz3.6.1

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​aa71e25

Merge remote-tracking branch 'trac/develop' into t/25090/public/pynormaliz1.17_normaliz3.6.1

[tscrim]

What is the status of this? Is it suppose to be in needs review? Is the empty polyhedron handled correctly now?

[mkoeppe]

Empty polyhedron is fixed. Needs review.

[tscrim]

LGTM overall, but a few methods are missing doctests:

• _init_from_normaliz_data
• _make_normaliz_cone

Also, INPUT: items generally do not end with a period.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7907f7a

Polyhedron_normaliz: Fix: INPUT: items generally do not end with a period

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1e20ef2

Polyhedron_normaliz._make_normaliz_cone: Add doctest

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f3d11a3

Polyhedron_normaliz._init_from_normaliz_data: Add doctest

[mkoeppe]

Ready for review

[tscrim]

Thank you. LGTM.

[mkoeppe]

Thanks for reviewing!

[vbraun]

sage -t --long src/sage/geometry/polyhedron/backend_normaliz.py
**********************************************************************
File "src/sage/geometry/polyhedron/backend_normaliz.py", line 451, in sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data
Failed example:
    P._get_nmzcone_data()
Exception raised:
    Traceback (most recent call last):
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 659, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1070, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data[1]>", line 1, in <module>
        P._get_nmzcone_data()
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.py", line 464, in _get_nmzcone_data
        import PyNormaliz
    ImportError: No module named PyNormaliz
**********************************************************************
File "src/sage/geometry/polyhedron/backend_normaliz.py", line 456, in sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data
Failed example:
    C = Polyhedron(backend='normaliz',rays=[[1,2],[2,1]])
Exception raised:
    Traceback (most recent call last):
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 659, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1070, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data[2]>", line 1, in <module>
        C = Polyhedron(backend='normaliz',rays=[[Integer(1),Integer(2)],[Integer(2),Integer(1)]])
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/constructor.py", line 600, in Polyhedron
        return parent(Vrep, Hrep, convert=convert, verbose=verbose)
      File "sage/structure/parent.pyx", line 923, in sage.structure.parent.Parent.__call__ (build/cythonized/sage/structure/parent.c:9706)
        return mor._call_with_args(x, args, kwds)
      File "sage/structure/coerce_maps.pyx", line 164, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_with_args (build/cythonized/sage/structure/coerce_maps.c:5112)
        raise
      File "sage/structure/coerce_maps.pyx", line 159, in sage.structure.coerce_maps.DefaultConvertMap_unique._call_with_args (build/cythonized/sage/structure/coerce_maps.c:5000)
        return C._element_constructor(x, *args, **kwds)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/parent.py", line 517, in _element_constructor_
        return self.element_class(self, Vrep, Hrep, **kwds)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.py", line 156, in __init__
        Polyhedron_base.__init__(self, parent, Vrep, Hrep, **kwds)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/base.py", line 124, in __init__
        self._init_from_Vrepresentation(vertices, rays, lines, **kwds)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.py", line 263, in _init_from_Vrepresentation
        self._init_from_normaliz_data(data, verbose=verbose)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/geometry/polyhedron/backend_normaliz.py", line 192, in _init_from_normaliz_data
        import PyNormaliz
    ImportError: No module named PyNormaliz
**********************************************************************
File "src/sage/geometry/polyhedron/backend_normaliz.py", line 457, in sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data
Failed example:
    C._get_nmzcone_data()
Exception raised:
    Traceback (most recent call last):
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 659, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1070, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data[3]>", line 1, in <module>
        C._get_nmzcone_data()
    NameError: name 'C' is not defined
**********************************************************************
1 item had failures:
   3 of   5 in sage.geometry.polyhedron.backend_normaliz.Polyhedron_normaliz._get_nmzcone_data
    [12 tests, 3 failures, 0.14 s]
----------------------------------------------------------------------
sage -t --long src/sage/geometry/polyhedron/backend_normaliz.py  # 3 doctests failed
----------------------------------------------------------------------

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​36d5473

src/sage/geometry/polyhedron/backend_normaliz.py: Add # optional - pynormaliz

[tscrim]

Whoops, forgot to test on a machine that did not have (py)normaliz installed. Sorry Volker.