spkg-configure.m4 for pari

[dimpase]

libpari-gmp is a dependency of many packages, so it's important to be able to use the one from the system.

We check that system's PARI/GP environment is close enough to what we would install in Sage (i.e. versions of GP and libpari, and that they share the datadir). Also, PARI packages need to be checked.

[dimpase]

we should use sample GP commands to test for packages, as in ​https://pari.math.u-bordeaux.fr/packages.html:

  ellinit("11a1")     /* package: elldata */
  galoisgetpol(8)     /* package: galpol */
  galoisgetname(12,1) /* package: galpol 4.0 and up */
  ellmodulareqn(11)   /* package: seadata */
  polgalois(x^8-2)    /* package: galdata */

if only the small version of seadata is installed, ellmodulareqn(211) produces an error (it does work with the "normal" seadata version).

need to figure out how to test for nftables.

[dimpase]

The following may be used to test whether libpari knows about galdata (Basically, obtained by editing the output of gp2c run on polygalois(x^8-2)[1]).

#include <pari/pari.h>  /* cc t.c -lpari && ./a.out */
GEN t_t(long prec)
{
  GEN x = pol_x(fetch_user_var("x"));
  return gel(polgalois(gsubgs(gpowgs(x, 8), 2), prec), 1);
}

int main() 
{
GEN r;
pari_init(5000000, 0);
r=t_t(100);
pari_printf("group order=%P4d\n", r); /* should be 16 */
pari_close();
return 0;
}

[dimpase]

New commits:

​83d26e5

spkg-configure for pari

[dimpase]

This is still WIP, more to be done here.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​4088bfc	spkg-configure for pari
​712fb95	spkg-configure for pari_elldata

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​2f45c9e	spkg-configure for pari
​b622942	spkg-configure for pari_elldata
​2e11aed	spkg-configure's for PARI packages

[dimpase]

still needed to sort out dependencies properly, and provide spkg-configure.m4 for gp2c.

For installing gp2c, one must provide pari.cfg location, something that it's not really discoverable. (save for find / -name pari.cfg :-))

[dimpase]

It seems that all the PARI package "deps" should be checked within spkg-configure of PARI itself (as PARI does not provide means to install extra packages, we need to disable system PARI lacking these packages). -- the branch does not yet implement this.

And not only standard packages pari_galdata and pari_seadata_small, but also the optional packages pari_elldata, pari_galpol, and pari_seadata.

The optional package pari_nftables is clearly an afterthought (not used anywhere in sagelib, too), and I don't want to rule out an otherwise OK installation of PARI/GP, is it is missing (e.g. Debian doesn't even ship it, although Fedora does ship it; not sure about Conda and Gentoo).

pari_jupyter should be installable regardless.

gp2c only needs the correct location of pari.cfg, and while it's a pain to find it, it is doable in some ad hoc way. I wonder what Conda does does with gp2c, and where it places pari.cfg.

[fbissey]

Distros have a bone to pick with the data packages from pari. That may be why some distro decided not to bother. There are ways to install the stuff manually if you want too.

Upstream is committing the cardinal sin of not versioning their data packages. The tarball is always called exactly the same - even when the content is changed. You understand that's a bit of a problem for automated build scripts that rely on a stored checksum to verify that the package is authentic.

For info, I had the stuff packaged in Gentoo and I recently updated it.

[dimpase]

I'm trying to convince the upstream to make configuration info for libpari discoverable. E.g. there is no way to get the location of pari.cfg from gp. I'm thinking of getting its location from the full path to libpari.so, I see no other good way...

[fbissey]

pari.cfg is installed in something called sysdatadir at configure time. I don't know if it is possible to get that variable at runtime (without knowing pari.cfg already that is).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​183f47c

check for PARI packages in PARI's spkg-configure

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​72f3c20

spkg-configure for gp2c

[isuruf]

Replying to dimpase:

And not only standard packages pari_galdata and pari_seadata_small, but also the optional packages pari_elldata, pari_galpol, and pari_seadata.

Is there a way to avoid checking for the optional packages? Maybe build PARI from SPKG if the optional package is not found, but requested.

The optional package pari_nftables is clearly an afterthought (not used anywhere in sagelib, too), and I don't want to rule out an otherwise OK installation of PARI/GP, is it is missing (e.g. Debian doesn't even ship it, although Fedora does ship it; not sure about Conda and Gentoo).

I just packaged all the optional PARI database packages on conda including nftables.

pari_jupyter should be installable regardless.

gp2c only needs the correct location of pari.cfg, and while it's a pain to find it, it is doable in some ad hoc way. I wonder what Conda does does with gp2c, and where it places pari.cfg.

Conda puts it into <prefix>/lib/pari/pari.cfg. We don't package gp2c yet.

[fbissey]

Someone put pari.cfg in a unhelpful location in Gentoo. In fact it broke my ebuild of gp2c. I don't know for how long but no one ever complained so I don't think there are many users for that program. Fixed in sage-on-gentoo's pari (2.11.1 and 2.11.2). Current sage-on-gentoo location is ${EPREFIX}/usr/share/pari/pari.cfg.

[dimpase]

by the way, it seems that cypari needs either a patch or an amendend spkg-install script. With the branch on this ticket, its installation fails with

Installing cypari-2.1.1
Installing package cypari2 using pip2
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-Nyv_mU
Created temporary directory: /tmp/pip-req-tracker-_Zruyw
Created requirements tracker '/tmp/pip-req-tracker-_Zruyw'
Created temporary directory: /tmp/pip-install-rsVw0C
Processing /home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src
  Created temporary directory: /tmp/pip-req-build-tpOOw3
  Added file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src to build tracker '/tmp/pip-req-tracker-_Zruyw'
  Running setup.py (path:/tmp/pip-req-build-tpOOw3/setup.py) egg_info for package from file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2
.1.1/src
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/cypari2.egg-info
    writing requirements to pip-egg-info/cypari2.egg-info/requires.txt
    writing pip-egg-info/cypari2.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/cypari2.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/cypari2.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/cypari2.egg-info/SOURCES.txt'
    Generating PARI functions: (!_) (#_) (%) (%#) (+_) (-_) Catalan/home/dimpase/sage/local/share/pari/doc not found at /usr/bin/gphelp line 147.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-tpOOw3/setup.py", line 83, in <module>
        cmdclass=dict(build_ext=build_ext, bdist_egg=no_egg)
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/__init__.py", line 143, in setup
        return distutils.core.setup(**attrs)
      File "/home/dimpase/sage/local/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/home/dimpase/sage/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/home/dimpase/sage/local/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 296, in run
        self.find_sources()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
        mm.run()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 534, in run
        self.add_defaults()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
        sdist.add_defaults(self)
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/home/dimpase/sage/local/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/home/dimpase/sage/local/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-req-build-tpOOw3/setup.py", line 33, in finalize_options
        rebuild()
      File "autogen/__init__.py", line 26, in rebuild
        G()
      File "autogen/generator.py", line 341, in __call__
        self.handle_pari_function(**v)
      File "autogen/generator.py", line 234, in handle_pari_function
        doc = get_rest_doc(function)
      File "autogen/doc.py", line 345, in get_rest_doc
        raw = get_raw_doc(function)
      File "autogen/doc.py", line 277, in get_raw_doc
        doc = subprocess.check_output(["gphelp", "-raw", function])
      File "/home/dimpase/sage/local/lib/python2.7/subprocess.py", line 223, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '[u'gphelp', u'-raw', u'Catalan']' returned non-zero exit status 2
Cleaning up...
  Removing source in /tmp/pip-req-build-tpOOw3
Removed file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src from build tracker '/tmp/pip-req-tracker-_Zruyw'
Removed build tracker '/tmp/pip-req-tracker-_Zruyw'
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-tpOOw3/

which appears to indicate that it does not try to figure out where PARI/GP docs are, but just assumes that they are in SAGE_LOCAL/share/pari/doc, instead of using `echo "default(datadir)" | gp -qf`/doc

[dimpase]

Oops, this is simply due to

# For PARI/GP
GP_DATA_DIR="$SAGE_LOCAL/share/pari" && export GP_DATA_DIR
GPHELP="$SAGE_LOCAL/bin/gphelp" && export GPHELP
GPDOCDIR="$SAGE_LOCAL/share/pari/doc" && export GPDOCDIR

in src/bin/sage-env.

OK, I can move this stuff into src/bin/sage-env-config(.in) and fill them in correctly.

[dimpase]

On my rather underpowered Debian laptop with all the PARI packages installed systemwide (i.e. all pari* SPKGs and gp2c are not built), this branch (with edits as in comment:17) builds and passes all the unrelated to dochtml tests, which building docs fails with multiprocessing-related segfault:

[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[dochtml]     "__main__", fname, loader, pkg_name)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/runpy.py", line 72, in _run_code
[dochtml]     exec code in run_globals
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 1718, in main
[dochtml]     builder()
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 354, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 550, in _wrapper
[dochtml]     build_many(build_ref_doc, L)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 288, in _build_many
[dochtml]     ret = x.get(99999)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/multiprocessing/pool.py", line 572, in get
[dochtml]     raise self._value
[dochtml] Exception: ('Non-exception during docbuild: Segmentation fault', SignalError('Segmentation fault',))
[dochtml] 
[dochtml]     Note: incremental documentation builds sometimes cause spurious
[dochtml]     error messages. To be certain that these are real errors, run
[dochtml]     "make doc-clean" first and try again.
make[3]: *** [Makefile:2032: doc-html] Error 1
make[3]: Leaving directory '/home/dimpase/sage/build/make'
make[2]: *** [Makefile:1843: all-start] Error 2
make[2]: Leaving directory '/home/dimpase/sage/build/make'

Any suggestions as to how to work around this?

[fbissey]

Do you have libpari-gmp or libpari-gmp-tls?

[dimpase]

I have libpari-gmp-tls

[dimpase]

Given that everything else works, I think it should be a docbuild bug, that produces segfaults at random places while reference manual is built. Something to do with tread-local storage in libpari, presumably.

[embray]

I know for pari we're also dependent currently on a few patches for bug fixes (on Cygwin one of these is critical). It would be nice to get these upstream if they aren't already...

[dimpase]

there is a Darwin-specific and a Cygwin-specific ones, the rest I presume are already in (in particular as they are by Pari people).

[dimpase]

Replying to dimpase:

Given that everything else works, I think it should be a docbuild bug, that produces segfaults at random places while reference manual is built. Something to do with tread-local storage in libpari, presumably.

this is most certainly #26608; I'll try an appropriately adjusted #27490 -workaround.

[dimpase]

Right, using the CYGWIN workaround, i.e. unconditional from .utils import _build_many as build_many in ​src/sage_setup/docbuild/__init__.py allows docs to build. Now, we can either always use it for external PARI, or test external PARI on whether it has multi-threading enabled, and only in this case use it.

I don't know how to test for PARI multi-threading enabled, so unless someone can provide a test, I'm going to set it on always.

[dimpase]

Also,

File "src/sage/misc/package.py", line 313, in sage.misc.package.is_package_installed
Failed example:
    is_package_installed('pari')  # optional - build
Expected:
    True
Got:
    False

needs a fix. (I guess dummy packages are not counted as installed).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6eac550

as pari might become dummy, test with GAP

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ac207d5

don't export GP_DATA_DIR, remove other, unneeded GP* vars

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​317f090

proper dependency resolution (all tests in PARI spkg-config)

[dimpase]

please review. I only tested this on Debian 10, which has all the require PARI packages. A new Fedora (29 or better) should do, too.

[dimpase]

The workaeound for docbuilding is not yet in this branch, thus if you supply a multithreaded libpari, docs won't build (unless one uses the workabout in comment:25)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2c26115

docbuild workaround for external PARI

[dimpase]

OK, so it should work with docbuilding too.

By the way, I'd also welcome review of autoconf/m4 code (e.g. m4/sage_check_pari_dep.m4 contains the 1st autoconf macro I ever wrote :-))

[isuruf]

I checked in conda and this picks up the conda installation correctly. For that though I needed to install the optional packages pari-*. conda install pari only pulls in pari-seadata-small and pari-galdata.

Even though the above issue is there, this is still an improvement over current situation.

[isuruf]

configure: === checking whether to install the pari SPKG ===
checking installing gmp/mpir or readline? ... no
checking installing PARI/GP packages? ... checking for gp... /projects/66d93023-00f0-4c12-8a25-5d6d4e486740/sage-build/bin/gp
checking is pari_elldata installed? ... yes
checking is pari_galdata installed? ... yes
checking is pari_galpol installed? ... yes
checking is pari_seadata installed? ...   ***   at top-level: poldegree(ellmodulareqn(211)[1])
  ***                           ^----------------------
  *** ellmodulareqn: error opening seadata file: `/projects/66d93023-00f0-4c12-8a25-5d6d4e486740/sage-build/share/pari/seadata/sea211'.
no; cannot use system pari/GP without seadata package
configure: Install seadata package and reconfigure.
configure: Otherwise Sage will build its own pari/GP.
using Sage's pari SPKG

Above error message from pari should go in config.log and not shown to the user.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​dd2b3fa

redirect gp errors to config.log

[dimpase]

OK, fixed.

[isuruf]

Thanks. Works for me

[fbissey]

Sorry for the late intervention.

diff --git a/src/sage_setup/docbuild/__init__.py b/src/sage_setup/docbuild/__init__.py
index e406bca..0e07c71 100644
--- a/src/sage_setup/docbuild/__init__.py
+++ b/src/sage_setup/docbuild/__init__.py
@@ -294,7 +294,7 @@ def _build_many(target, args):
             raise
     return ret
 
-if not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3):
+if (os.environ['SAGE_PARI_CFG'] !='') and (not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3)):
     build_many = _build_many
 else:
     # Cygwin 64-bit < 3.0.0 has a bug with exception handling when exceptions

That bit causes me trouble when building the doc

 * python3_7: running distutils-r1_run_phase python_compile_all
Setting permissions of DOT_SAGE directory so only you can read and write it.
Traceback (most recent call last):
  File "sage_setup/docbuild/__main__.py", line 1, in <module>
    from sage_setup.docbuild import main
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 297, in <module>
    if (os.environ['SAGE_PARI_CFG'] !='') and (not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3)):
  File "/usr/lib/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'SAGE_PARI_CFG'

[fbissey]

I would be happy for this ticket to proceed as is and the issue to be addressed in a follow up ticket.

But the general assumption should be sage-on-distros don't run configure. In fact we have pushed a few things so that sage-distro can run without sage-env. I did so a few release ago and Antonio Rojas from arch has pushed so he could follow my lead there in sage-8.9. We can define the variable at build time, I could even flip the value depending on whether pari has been built with tls in Gentoo. But it would be cleaner if it just worked, with the assumption that pari is provided externally, if the variable is undefined.

[fbissey]

Follow up will take place at #28341.

[dimpase]

Can Archlinux get more PARI packages (as much as Debian 10), so that its PARI may be used?

[arojas]

Replying to dimpase:

Can Archlinux get more PARI packages (as much as Debian 10), so that its PARI may be used?

Standard packages are available from the official repositories, optional packages can easily be installed from AUR - ​https://aur.archlinux.org/packages/?K=pari-

[mkoeppe]

Follow up at #29319

[mkoeppe]

Replying to dimpase:

by the way, it seems that cypari needs either a patch or an amendend spkg-install script. With the branch on this ticket, its installation fails with

Installing cypari-2.1.1
...
      File "autogen/doc.py", line 277, in get_raw_doc
        doc = subprocess.check_output(["gphelp", "-raw", function])
      File "/home/dimpase/sage/local/lib/python2.7/subprocess.py", line 223, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '[u'gphelp', u'-raw', u'Catalan']' returned non-zero exit status 2 ...

which appears to indicate that it does not try to figure out where PARI/GP docs are, but just assumes that they are in SAGE_LOCAL/share/pari/doc, instead of using `echo "default(datadir)" | gp -qf`/doc

Follow up for this problem: #29342

[gh-thierry-FreeBSD]

I was working on FreeBSD with 9.1.beta9 and pari as a system package without any problem related to gp / pari.

Yesterday I switched to 9.1.rc1, with the same external package, and now the compilation of cysignals aborts with this message:

[cysignals-1.10.2]     cc -pthread -shared -L/usr/local/lib -fstack-protector-strong -L/usr/ports/math/sage/work/stage/usr/local/lib -Wl,-rpath,/usr/ports/math/sage/work/stage/usr/local/lib -L/usr/local/llvm90/lib -L/usr/local/lib -Wl,-rpath=/usr/local/lib/gcc9 -L/usr/local/lib/gcc9 -B/usr/local/bin -L/usr/local/lib -fstack-protector-strong -O2 -pipe -DLIBICONV_PLUG -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -DLIBICONV_PLUG -I/usr/local/include -isystem /usr/local/include -Wp,-U_FORTIFY_SOURCE build/temp.freebsd-12.1-STABLE-amd64-3.7/build/src/cysignals/signals.o -L/usr/local/lib -lpython3.7m -o build/lib.freebsd-12.1-STABLE-amd64-3.7/cysignals/signals.so -lpari -lomp -pthread -L/usr/local/lib
[cysignals-1.10.2]     /usr/local/bin/ld: PARI_SIGINT_block: TLS definition in /usr/local/lib/libpari.so section .tbss mismatches non-TLS reference in build/temp.freebsd-12.1-STABLE-amd64-3.7/build/src/cysignals/signals.o
[cysignals-1.10.2]     /usr/local/bin/ld: /usr/local/lib/libpari.so: error adding symbols: bad value

Anyone with the same problem on other OS, or is it specific to FreeBSD?

[dimpase]

Hmm, not sure what's happening here. I don't think I saw this before. Something related to --enable-tls (tls=thread-local storage) option while buildig Pari/GP - which works on Linux smoothly.

[gh-thierry-FreeBSD]

But it seems that this is disabled when Sage builds Pari?

In build/pkgs/pari/spkg-install.in there is "unset enable_tls".

[dimpase]

I mean to say that we can use Pari built with --enable-tls with Sage on several Linux distributions (not the Pari built by Sage). Does FreeBSD Pari port use some special clang flags for TLS?

Is Pari built with pthread on FreeBSD?

[dimpase]

On the other hand, cysignals is not known to be thread-safe, so it could be that FreeBSD tools are better than Linux tools in detecting this: ​https://github.com/sagemath/cysignals/issues/21

In view of this, linking cysignals with -pthreads might be what causes this error.

[gh-thierry-FreeBSD]

Actually the problem was different! Pari's spkg-configure.m4 is broken: I thought that I was using pari from the system packages, but I was not.

spkg-configure.m4 includes unescaped [ ] and it produces an error.

A fix could be:

--- build/pkgs/pari/spkg-configure.m4.orig      2020-04-27 19:20:17.006373000 +0200
+++ build/pkgs/pari/spkg-configure.m4   2020-04-27 19:34:16.220830000 +0200
@@ -74,7 +74,7 @@
             sage_spkg_install_pari=yes
         fi
         AC_MSG_CHECKING([whether hyperellcharpoly bug is fixed])
-        bug_check=$(echo "hyperellcharpoly([x^10 + x^9 + x^8 + x,0]*Mod(1,3))" | $GP -qf 2>> config.log)
+        bug_check=$(echo "hyperellcharpoly([[x^10 + x^9 + x^8 + x,0]]*Mod(1,3))" | $GP -qf 2>> config.log)
         expected="%1 = x^8 + 731*x^7 + 6*x^6 - 720*x^5 + 18*x^4 - 2160*x^3 + 54*x^2 + 19737*x + 81"
         if test x"$bug_check" = x"$expected"; then
            AC_MSG_RESULT([yes])

(But ATM I removed this part)

[mkoeppe]

That's fixed already by #29554

[dimpase]

Replying to mkoeppe:

That's fixed already by #29554

however, this still begs a question why cysignals is built with -pythread (as set in its setup.py), while it's known not to be thread-safe, cf. ​https://github.com/sagemath/cysignals/issues/21