Create a variant of the polymake interface using polymake's callable library (polymake::Main) via JuPyMake

[mkoeppe]

Following up on #22452 (Create a Polymake pexpect interface):

As suggested by Gawrilow in ​https://forum.polymake.org/viewtopic.php?f=8&t=544, we could create a variant of the polymake pexpect interface that does not go through pty's but rather replaces the relevant send/receive methods of pexpect by calls to simulate_shell_input (linking to libpolymake).

Recent polymake versions have developed this further and now provide a class polymake::Main with convenient access to the polymake interpreter. We use the (partial) Python bindings for this, ​https://github.com/polymake/JuPyMake which are used for Polymake's jupyter.

Upstream package:

  wget -P upstream https://files.pythonhosted.org/packages/32/dd/0a48546daa2461e57207d31eeb64a3ad77fd9fb339d90e1e067ea25991a5/JuPyMake-0.9.tar.gz

We do want to keep the current pty-based pexpect interface, though, because it may be handy for running polymake on a remote server; or for guarding against a crashing polymake (sigh).

Remaining issue: #27807

See also: #22710: Meta-ticket: polymake

[mkoeppe]

Here's a first version.

---

Last 10 new commits:

​7018013	polymake doctest: Mark a doctest random that depends on installed packages
​78c10fb	Polymake._tab_completion: Fix for new apropos format in Polymake 3.4
​1699019	Doctest fix for polymake 3.4: false/true instead of 0/1
​f595909	Doctest fix for polymake 3.4: Use different function from the tropical application
​76701f6	Doctest fix for polymake 3.4: Differences in available properties
​83b51de	Polymake 3.4: Fix for typeof small objects
​8510124	Polymake: Remove copy/paste error handling for code that cannot raise
​3c6ae95	polymake: Make normaliz a dependency
​b6eeaa1	polymake: Add a patch to polymake's libnormaliz configure
​2f855b0	Refactor interface Polymake through PolymakeAbstract, add PolymakeJuPyMake

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​78b5036

More doctests etc.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​92274e0

Add package jupymake

[mkoeppe]

Some polymake errors are not properly signalled by Polymake: Only the stderr slot [2] is filled but the error slot [3] is empty

sage: from JuPyMake import ExecuteCommand
sage: ExecuteCommand('help("Triangulation");')
(True, '', "polymake:  ERROR: unknown help topic 'Triangulation'\n", '')

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b40a1cb	No need for File::Slurp in jupymake interface
​6bd4971	Add doc

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0bad10c

Fix jupymake help

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​624ac01	polymake: Also provide core functions
​6708663	Merge branch 't/24905/public/upgrade_polymake_to_version_3_2r4' into t/22704/public/22704

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a5c8eb8	Add optional - polymake
​ad2b976	Can only quit polymake in pexpect interface
​7ec6c74	Mark jupymake tests optional - jupymake
​8b7ffbd	Select polymake interface based on feature test
​a614876	Fixup polymake doctests that depend on 'used package' messages

[mkoeppe]

Signal handling issues are solved in new version of JuPyMake? - waiting for a new release

[jipilab]

This is coming from #24905, which is already set as positive review.

Could you correct the following pyflakes warnings:

+src/sage/geometry/polyhedron/backend_polymake.py:29: 'sage.rings.all.QQ' imported but unused
+src/sage/geometry/polyhedron/backend_polymake.py:29: 'sage.rings.all.ZZ' imported but unused

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​bcc7c6b

Add polymake patch for thread correctness

[mkoeppe]

Crashes are fixed.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​34900a0	backend_polymake.py: Remove unused imports
​1b8e2a5	Update jupymake to 0.9

[mkoeppe]

Signal handling fixed by jupymake update

[mkoeppe]

polymake's error handling bug is reported upstream.

[mkoeppe]

Apart from 1 remaining doctest regarding the form of some error messages, this is ready for review!

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​666db96	Added two missing optional tags
​4e91dcc	Change format of a 'not tested' doc test annotation
​05f3e61	update docs, as discussed on #24905
​53c8ee6	Mention libterm-readkey-perl
​b613e57	Mention libterm-readline-gnu-perl
​e5c3249	polymake: Mention MOngoDB for polydb
​b89376a	polymake: Update info on macOS
​0c1b044	polymake: update MongoDB information for linux distros
​e46cd38	Merge branch 't/24905/public/upgrade_polymake_to_version_3_2r4' into t/22704/public/22704

[dimpase]

does this need #24905 ?

[mkoeppe]

I did this work on top of the #24905 branch because otherwise there would be big merge conflicts.

[dimpase]

that's what I get after trying this on Linux:

$ ./sage -tp --optional=bliss,build,dochtml,e_antic,gap_packages,jupymake,lidia,lrslib,memlimit,ninja_build,normaliz,python2,polymake,sage src/sage/interfaces/polymake.py 
too many failed tests, not using stored timings
Running doctests with ID 2019-05-08-22-08-17-11941b32.
Git branch: testwip
Using --optional=bliss,build,dochtml,e_antic,gap_packages,jupymake,lidia,lrslib,memlimit,ninja_build,normaliz,polymake,python2,sage
Doctesting 1 file using 4 threads.
sage -t src/sage/interfaces/polymake.py
**********************************************************************
File "src/sage/interfaces/polymake.py", line 765, in sage.interfaces.polymake.PolymakeAbstract.application
Failed example:
    polymake.eval('application "killerapp";')         # optional - polymake
Expected:
    Traceback (most recent call last):
    ...
    PolymakeError: Unknown application killerapp
Got:
    <BLANKLINE>
    Traceback (most recent call last):
      File "/mnt/opt/Sage/sage-dev/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 671, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/mnt/opt/Sage/sage-dev/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1095, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.interfaces.polymake.PolymakeAbstract.application[15]>", line 1, in <module>
        polymake.eval('application "killerapp";')         # optional - polymake
      File "/mnt/opt/Sage/sage-dev/local/lib/python2.7/site-packages/sage/interfaces/polymake.py", line 2520, in eval
        raise PolymakeError(error)
    PolymakeError: Can't locate object method "compile_scope" via package "Polymake::Core::Shell::Mock" at /mnt/opt/Sage/sage-dev/local/share/polymake/perllib/Polymake/utils.pl line 247.
    <BLANKLINE>
**********************************************************************
1 item had failures:
   1 of  17 in sage.interfaces.polymake.PolymakeAbstract.application
    [317 tests, 1 failure, 9.54 s]
----------------------------------------------------------------------
sage -t src/sage/interfaces/polymake.py  # 1 doctest failed
----------------------------------------------------------------------

[dimpase]

and also

**********************************************************************
File "src/sage/geometry/polyhedron/base.py", line 175, in sage.geometry.polyhedron.base.Polyhedron_base._sage_input_
Failed example:
    P = Polyhedron(vertices = [[1, 0], [0, 1]], rays = [[1, 1]], backend='polymake') # optional - polymake
Expected nothing
Got:
     gmp v.6.1
...
File "src/sage/geometry/polyhedron/backend_polymake.py", line 59, in sage.geometry.polyhedron.backend_polymake.Polyhedron_polymake
Failed example:
    p = Polyhedron(vertices=[(0,0),(1,0),(0,1)], rays=[(1,1)],   # optional - polymake
                   lines=[], backend='polymake')
Expected nothing
Got:
     gmp v.6.1

[mkoeppe]

Replying to dimpase:

that's what I get after trying this on Linux:

$ ./sage -tp --optional=bliss,build,dochtml,e_antic,gap_packages,jupymake,lidia,lrslib,memlimit,ninja_build,normaliz,python2,polymake,sage src/sage/interfaces/polymake.py 
....
       raise PolymakeError(error)
    PolymakeError: Can't locate object method "compile_scope" via package "Polymake::Core::Shell::Mock" at /mnt/opt/Sage/sage-dev/local/share/polymake/perllib/Polymake/utils.pl line 247.

Yes, I know this one - see ticket description. I have reported it upstream.

[mkoeppe]

Replying to dimpase:

and also

**********************************************************************
File "src/sage/geometry/polyhedron/base.py", line 175, in sage.geometry.polyhedron.base.Polyhedron_base._sage_input_
Failed example:
    P = Polyhedron(vertices = [[1, 0], [0, 1]], rays = [[1, 1]], backend='polymake') # optional - polymake
Expected nothing
Got:
     gmp v.6.1
...
File "src/sage/geometry/polyhedron/backend_polymake.py", line 59, in sage.geometry.polyhedron.backend_polymake.Polyhedron_polymake
Failed example:
    p = Polyhedron(vertices=[(0,0),(1,0),(0,1)], rays=[(1,1)],   # optional - polymake
                   lines=[], backend='polymake')
Expected nothing
Got:
     gmp v.6.1

Looks like this message comes from lrslib, I'll investigate

[mkoeppe]

Actually on my machine polymake does not find the installed lrslib (it's looking for liblrs but we only have liblrsgmp.)

[dimpase]

I also only have liblrsgmp. Not sure if the following helps, but still:

$ ldd local/lib/libpolymake.so
	linux-vdso.so.1 (0x00007fff97bbb000)
	libperl.so.5.28 => /usr/lib64/libperl.so.5.28 (0x00007fcae6d94000)
	libxml2.so.2 => /usr/lib64/libxml2.so.2 (0x00007fcae6bef000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fcae6be9000)
	libmpfr.so.6 => /mnt/opt/Sage/sage-dev/local/lib/libmpfr.so.6 (0x00007fcae6b6a000)
	libgmp.so.10 => /usr/lib64/libgmp.so.10 (0x00007fcae68df000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fcae68bc000)
	libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/libstdc++.so.6 (0x00007fcae6642000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fcae6505000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/9.1.0/libgcc_s.so.1 (0x00007fcae64eb000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fcae631b000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fcae62e2000)
	libicuuc.so.64 => /usr/lib64/libicuuc.so.64 (0x00007fcae60f1000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fcae5ecf000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fcae71bd000)
	libicudata.so.64 => /usr/lib64/libicudata.so.64 (0x00007fcae4489000)

the rest of polymake libs are statically linked, so it's not as easy to gather whether lrs has anything to do with it.

[mkoeppe]

Replying to mkoeppe:

Actually on my machine polymake does not find the installed lrslib (it's looking for liblrs but we only have liblrsgmp.)

This belongs to the polymake 3.4 upgrade ticket (#24905).

[dimpase]

Replying to mkoeppe:

Replying to dimpase:

and also

**********************************************************************
File "src/sage/geometry/polyhedron/base.py", line 175, in sage.geometry.polyhedron.base.Polyhedron_base._sage_input_
Failed example:
    P = Polyhedron(vertices = [[1, 0], [0, 1]], rays = [[1, 1]], backend='polymake') # optional - polymake
Expected nothing
Got:
     gmp v.6.1
...
File "src/sage/geometry/polyhedron/backend_polymake.py", line 59, in sage.geometry.polyhedron.backend_polymake.Polyhedron_polymake
Failed example:
    p = Polyhedron(vertices=[(0,0),(1,0),(0,1)], rays=[(1,1)],   # optional - polymake
                   lines=[], backend='polymake')
Expected nothing
Got:
     gmp v.6.1

Looks like this message comes from lrslib, I'll investigate

After the lrs fix on #27803, one of these 2 errors is still present:

File "src/sage/geometry/polyhedron/base.py", line 175, in sage.geometry.polyhedron.base.Polyhedron_base._sage_input_
Failed example:
    P = Polyhedron(vertices = [[1, 0], [0, 1]], rays = [[1, 1]], backend='polymake') # optional - polymake
Expected nothing
Got:
     gmp v.6.1

[mkoeppe]

Yes, and with #27803 I can reproduce it. I think these errors appear non-deterministicaly.

[dimpase]

Replying to mkoeppe:

Yes, and with #27803 I can reproduce it. I think these errors appear non-deterministicaly.

I think it is deterministic, and occurs on the 1st invocation of the polymake interface. Try starting Sage and running

P = Polyhedron(vertices = [[1, 0], [0, 1]], rays = [[1, 1]], backend='polymake')

repeatedly.

[mkoeppe]

Replying to dimpase:

I think it is deterministic, and occurs on the 1st invocation of the polymake interface.

You are right.

[mkoeppe]

The new behavior from this ticket comes from the fact that with the new in-process polymake, lrslib prints directly to stdout, so this output is not captured. Perhaps lrslib should be configured with LRS_QUIET.

[mkoeppe]

Replying to mkoeppe:

Perhaps lrslib should be configured with LRS_QUIET.

lrslib 070 removes this output unconditionally.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2eda6e1	polymake: add to fedora instructions
​5916141	Fix polymake lrslib detection
​086bb2d	Merge branch 't/27803/public/27803-polymake-lrslib-detection' into t/22704/public/22704
​1b47911	Configure lrslib with LRS_QUIET
​c84d89d	Merge branch 't/27805/configure_lrslib_with_lrs_quiet' into t/22704/public/22704

[mkoeppe]

The lrslib 070 update (#27745) is too much work for today. Instead I configure lrslib with LRS_QUIET (#27805).

[dimpase]

OK, this is fixed by #27805 indeed.

---

New commits:

​2eda6e1	polymake: add to fedora instructions
​5916141	Fix polymake lrslib detection
​086bb2d	Merge branch 't/27803/public/27803-polymake-lrslib-detection' into t/22704/public/22704
​1b47911	Configure lrslib with LRS_QUIET
​c84d89d	Merge branch 't/27805/configure_lrslib_with_lrs_quiet' into t/22704/public/22704

[mkoeppe]

I've put the remaining minor issue to a follow-up ticket, #27807 (Polymake-jupymake interface: Fix polymake's error handling in Shell::Mock)

[mkoeppe]

Should I make that last failing doctest less specific regarding the error message for this ticket? (Before it's fixed fully in #27807)

[dimpase]

If you like, you can tag it "known error".

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​58aa1ad

Mark issue of #27807 as known bug

[vbraun]

Merge conflict

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8d9e2c2

Merge tag '8.8.beta5' into t/22704/public/22704

[mkoeppe]

The upstream package jupymake seems to be missing on the sage mirrors

[jipilab]

Do I understand correctly that in the current version jupymake is necessary for polymake to function properly? Shouldn't jupymake become a dependancy of polymake?

[mkoeppe]

Replying to jipilab:

Do I understand correctly that in the current version jupymake is necessary for polymake to function properly? Shouldn't jupymake become a dependancy of polymake?

No, there are two interface classes, which both work. PolymakeExpect (aliases to Polymake) and PolymakeJupyMake. Depending on whether you have jupymake, polymake is set to an instance of one or the other.