Opened 4 months ago

Closed 3 months ago

Last modified 3 months ago

#28242 closed enhancement (fixed)

spkg-configure.m4 for pari

Reported by: dimpase Owned by:
Priority: major Milestone: sage-8.9
Component: build: configure Keywords: spkg-configure
Cc: embray, fbissey, isuruf, gh-timokau, arojas Merged in:
Authors: Dima Pasechnik Reviewers: Isuru Fernando
Report Upstream: N/A Work issues:
Branch: dd2b3fa (Commits) Commit:
Dependencies: Stopgaps:

Description (last modified by dimpase)

libpari-gmp is a dependency of many packages, so it's important to be able to use the one from the system.

We check that system's PARI/GP environment is close enough to what we would install in Sage (i.e. versions of GP and libpari, and that they share the datadir). Also, PARI packages need to be checked.

Change History (45)

comment:1 Changed 4 months ago by dimpase

we should use sample GP commands to test for packages, as in https://pari.math.u-bordeaux.fr/packages.html:

  ellinit("11a1")     /* package: elldata */
  galoisgetpol(8)     /* package: galpol */
  galoisgetname(12,1) /* package: galpol 4.0 and up */
  ellmodulareqn(11)   /* package: seadata */
  polgalois(x^8-2)    /* package: galdata */

if only the small version of seadata is installed, ellmodulareqn(211) produces an error (it does work with the "normal" seadata version).

need to figure out how to test for nftables.

comment:2 Changed 4 months ago by dimpase

The following may be used to test whether libpari knows about galdata (Basically, obtained by editing the output of gp2c run on polygalois(x^8-2)[1]).

#include <pari/pari.h>  /* cc t.c -lpari && ./a.out */
GEN t_t(long prec)
{
  GEN x = pol_x(fetch_user_var("x"));
  return gel(polgalois(gsubgs(gpowgs(x, 8), 2), prec), 1);
}

int main() 
{
GEN r;
pari_init(5000000, 0);
r=t_t(100);
pari_printf("group order=%P4d\n", r); /* should be 16 */
pari_close();
return 0;
}

comment:3 Changed 4 months ago by dimpase

  • Authors set to Dima Pasechnik
  • Branch set to u/dimpase/packages/pariconfig
  • Commit set to 83d26e536c4d8e0f75e50a0c22e5d3c736e3aef8
  • Description modified (diff)

New commits:

83d26e5spkg-configure for pari

comment:4 Changed 4 months ago by dimpase

This is still WIP, more to be done here.

comment:5 Changed 4 months ago by git

  • Commit changed from 83d26e536c4d8e0f75e50a0c22e5d3c736e3aef8 to 712fb952c1bee5fefe3ec98781d397f9ecbbb837

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

4088bfcspkg-configure for pari
712fb95spkg-configure for pari_elldata

comment:6 Changed 4 months ago by git

  • Commit changed from 712fb952c1bee5fefe3ec98781d397f9ecbbb837 to 2e11aed9dfd4572b0185c727509fb31b765a95d2

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

2f45c9espkg-configure for pari
b622942spkg-configure for pari_elldata
2e11aedspkg-configure's for PARI packages

comment:7 Changed 4 months ago by dimpase

  • Cc embray added

still needed to sort out dependencies properly, and provide spkg-configure.m4 for gp2c.

For installing gp2c, one must provide pari.cfg location, something that it's not really discoverable. (save for find / -name pari.cfg :-))

comment:8 follow-up: Changed 4 months ago by dimpase

  • Cc fbissey isuruf added

It seems that all the PARI package "deps" should be checked within spkg-configure of PARI itself (as PARI does not provide means to install extra packages, we need to disable system PARI lacking these packages). -- the branch does not yet implement this.

And not only standard packages pari_galdata and pari_seadata_small, but also the optional packages pari_elldata, pari_galpol, and pari_seadata.

The optional package pari_nftables is clearly an afterthought (not used anywhere in sagelib, too), and I don't want to rule out an otherwise OK installation of PARI/GP, is it is missing (e.g. Debian doesn't even ship it, although Fedora does ship it; not sure about Conda and Gentoo).

pari_jupyter should be installable regardless.

gp2c only needs the correct location of pari.cfg, and while it's a pain to find it, it is doable in some ad hoc way. I wonder what Conda does does with gp2c, and where it places pari.cfg.

comment:9 Changed 4 months ago by fbissey

Distros have a bone to pick with the data packages from pari. That may be why some distro decided not to bother. There are ways to install the stuff manually if you want too.

Upstream is committing the cardinal sin of not versioning their data packages. The tarball is always called exactly the same - even when the content is changed. You understand that's a bit of a problem for automated build scripts that rely on a stored checksum to verify that the package is authentic.

For info, I had the stuff packaged in Gentoo and I recently updated it.

comment:10 Changed 4 months ago by dimpase

I'm trying to convince the upstream to make configuration info for libpari discoverable. E.g. there is no way to get the location of pari.cfg from gp. I'm thinking of getting its location from the full path to libpari.so, I see no other good way...

comment:11 Changed 4 months ago by fbissey

pari.cfg is installed in something called sysdatadir at configure time. I don't know if it is possible to get that variable at runtime (without knowing pari.cfg already that is).

comment:12 Changed 4 months ago by git

  • Commit changed from 2e11aed9dfd4572b0185c727509fb31b765a95d2 to 183f47c23e5b8a3f1acdf37b9c2c37f8cd337ee5

Branch pushed to git repo; I updated commit sha1. New commits:

183f47ccheck for PARI packages in PARI's spkg-configure

comment:13 Changed 4 months ago by git

  • Commit changed from 183f47c23e5b8a3f1acdf37b9c2c37f8cd337ee5 to 72f3c20b38dd0aad6c8325f16f3b4be79b7c70ee

Branch pushed to git repo; I updated commit sha1. New commits:

72f3c20spkg-configure for gp2c

comment:14 in reply to: ↑ 8 Changed 4 months ago by isuruf

Replying to dimpase:

And not only standard packages pari_galdata and pari_seadata_small, but also the optional packages pari_elldata, pari_galpol, and pari_seadata.

Is there a way to avoid checking for the optional packages? Maybe build PARI from SPKG if the optional package is not found, but requested.

The optional package pari_nftables is clearly an afterthought (not used anywhere in sagelib, too), and I don't want to rule out an otherwise OK installation of PARI/GP, is it is missing (e.g. Debian doesn't even ship it, although Fedora does ship it; not sure about Conda and Gentoo).

I just packaged all the optional PARI database packages on conda including nftables.

pari_jupyter should be installable regardless.

gp2c only needs the correct location of pari.cfg, and while it's a pain to find it, it is doable in some ad hoc way. I wonder what Conda does does with gp2c, and where it places pari.cfg.

Conda puts it into <prefix>/lib/pari/pari.cfg. We don't package gp2c yet.

Last edited 4 months ago by isuruf (previous) (diff)

comment:15 Changed 4 months ago by fbissey

Someone put pari.cfg in a unhelpful location in Gentoo. In fact it broke my ebuild of gp2c. I don't know for how long but no one ever complained so I don't think there are many users for that program. Fixed in sage-on-gentoo's pari (2.11.1 and 2.11.2). Current sage-on-gentoo location is ${EPREFIX}/usr/share/pari/pari.cfg.

comment:16 Changed 4 months ago by dimpase

by the way, it seems that cypari needs either a patch or an amendend spkg-install script. With the branch on this ticket, its installation fails with

Installing cypari-2.1.1
Installing package cypari2 using pip2
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-ephem-wheel-cache-Nyv_mU
Created temporary directory: /tmp/pip-req-tracker-_Zruyw
Created requirements tracker '/tmp/pip-req-tracker-_Zruyw'
Created temporary directory: /tmp/pip-install-rsVw0C
Processing /home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src
  Created temporary directory: /tmp/pip-req-build-tpOOw3
  Added file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src to build tracker '/tmp/pip-req-tracker-_Zruyw'
  Running setup.py (path:/tmp/pip-req-build-tpOOw3/setup.py) egg_info for package from file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2
.1.1/src
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/cypari2.egg-info
    writing requirements to pip-egg-info/cypari2.egg-info/requires.txt
    writing pip-egg-info/cypari2.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/cypari2.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/cypari2.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/cypari2.egg-info/SOURCES.txt'
    Generating PARI functions: (!_) (#_) (%) (%#) (+_) (-_) Catalan/home/dimpase/sage/local/share/pari/doc not found at /usr/bin/gphelp line 147.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-tpOOw3/setup.py", line 83, in <module>
        cmdclass=dict(build_ext=build_ext, bdist_egg=no_egg)
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/__init__.py", line 143, in setup
        return distutils.core.setup(**attrs)
      File "/home/dimpase/sage/local/lib/python2.7/distutils/core.py", line 151, in setup
        dist.run_commands()
      File "/home/dimpase/sage/local/lib/python2.7/distutils/dist.py", line 953, in run_commands
        self.run_command(cmd)
      File "/home/dimpase/sage/local/lib/python2.7/distutils/dist.py", line 972, in run_command
        cmd_obj.run()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 296, in run
        self.find_sources()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 303, in find_sources
        mm.run()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 534, in run
        self.add_defaults()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/egg_info.py", line 570, in add_defaults
        sdist.add_defaults(self)
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/home/dimpase/sage/local/lib/python2.7/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/home/dimpase/sage/local/lib/python2.7/distutils/cmd.py", line 312, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/home/dimpase/sage/local/lib/python2.7/distutils/cmd.py", line 109, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-req-build-tpOOw3/setup.py", line 33, in finalize_options
        rebuild()
      File "autogen/__init__.py", line 26, in rebuild
        G()
      File "autogen/generator.py", line 341, in __call__
        self.handle_pari_function(**v)
      File "autogen/generator.py", line 234, in handle_pari_function
        doc = get_rest_doc(function)
      File "autogen/doc.py", line 345, in get_rest_doc
        raw = get_raw_doc(function)
      File "autogen/doc.py", line 277, in get_raw_doc
        doc = subprocess.check_output(["gphelp", "-raw", function])
      File "/home/dimpase/sage/local/lib/python2.7/subprocess.py", line 223, in check_output
        raise CalledProcessError(retcode, cmd, output=output)
    subprocess.CalledProcessError: Command '[u'gphelp', u'-raw', u'Catalan']' returned non-zero exit status 2
Cleaning up...
  Removing source in /tmp/pip-req-build-tpOOw3
Removed file:///home/dimpase/sage/local/var/tmp/sage/build/cypari-2.1.1/src from build tracker '/tmp/pip-req-tracker-_Zruyw'
Removed build tracker '/tmp/pip-req-tracker-_Zruyw'
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-tpOOw3/

which appears to indicate that it does not try to figure out where PARI/GP docs are, but just assumes that they are in SAGE_LOCAL/share/pari/doc, instead of using `echo "default(datadir)" | gp -qf`/doc

comment:17 Changed 4 months ago by dimpase

Oops, this is simply due to

# For PARI/GP
GP_DATA_DIR="$SAGE_LOCAL/share/pari" && export GP_DATA_DIR
GPHELP="$SAGE_LOCAL/bin/gphelp" && export GPHELP
GPDOCDIR="$SAGE_LOCAL/share/pari/doc" && export GPDOCDIR

in src/bin/sage-env.

OK, I can move this stuff into src/bin/sage-env-config(.in) and fill them in correctly.

comment:18 Changed 4 months ago by dimpase

On my rather underpowered Debian laptop with all the PARI packages installed systemwide (i.e. all pari* SPKGs and gp2c are not built), this branch (with edits as in comment:17) builds and passes all the unrelated to dochtml tests, which building docs fails with multiprocessing-related segfault:

[dochtml] Error building the documentation.
[dochtml] Traceback (most recent call last):
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
[dochtml]     "__main__", fname, loader, pkg_name)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/runpy.py", line 72, in _run_code
[dochtml]     exec code in run_globals
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__main__.py", line 2, in <module>
[dochtml]     main()
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 1718, in main
[dochtml]     builder()
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 354, in _wrapper
[dochtml]     getattr(get_builder(document), 'inventory')(*args, **kwds)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 550, in _wrapper
[dochtml]     build_many(build_ref_doc, L)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/site-packages/sage_setup/docbuild/__init__.py", line 288, in _build_many
[dochtml]     ret = x.get(99999)
[dochtml]   File "/home/dimpase/sage/local/lib/python2.7/multiprocessing/pool.py", line 572, in get
[dochtml]     raise self._value
[dochtml] Exception: ('Non-exception during docbuild: Segmentation fault', SignalError('Segmentation fault',))
[dochtml] 
[dochtml]     Note: incremental documentation builds sometimes cause spurious
[dochtml]     error messages. To be certain that these are real errors, run
[dochtml]     "make doc-clean" first and try again.
make[3]: *** [Makefile:2032: doc-html] Error 1
make[3]: Leaving directory '/home/dimpase/sage/build/make'
make[2]: *** [Makefile:1843: all-start] Error 2
make[2]: Leaving directory '/home/dimpase/sage/build/make'

Any suggestions as to how to work around this?

comment:19 Changed 4 months ago by fbissey

Do you have libpari-gmp or libpari-gmp-tls?

comment:20 Changed 4 months ago by dimpase

I have libpari-gmp-tls

comment:21 follow-up: Changed 4 months ago by dimpase

Given that everything else works, I think it should be a docbuild bug, that produces segfaults at random places while reference manual is built. Something to do with tread-local storage in libpari, presumably.

comment:22 Changed 4 months ago by embray

I know for pari we're also dependent currently on a few patches for bug fixes (on Cygwin one of these is critical). It would be nice to get these upstream if they aren't already...

comment:23 Changed 4 months ago by dimpase

there is a Darwin-specific and a Cygwin-specific ones, the rest I presume are already in (in particular as they are by Pari people).

comment:24 in reply to: ↑ 21 Changed 4 months ago by dimpase

Replying to dimpase:

Given that everything else works, I think it should be a docbuild bug, that produces segfaults at random places while reference manual is built. Something to do with tread-local storage in libpari, presumably.

this is most certainly #26608; I'll try an appropriately adjusted #27490 -workaround.

comment:25 Changed 4 months ago by dimpase

Right, using the CYGWIN workaround, i.e. unconditional from .utils import _build_many as build_many in src/sage_setup/docbuild/__init__.py allows docs to build. Now, we can either always use it for external PARI, or test external PARI on whether it has multi-threading enabled, and only in this case use it.

I don't know how to test for PARI multi-threading enabled, so unless someone can provide a test, I'm going to set it on always.

comment:26 Changed 4 months ago by dimpase

Also,

File "src/sage/misc/package.py", line 313, in sage.misc.package.is_package_installed
Failed example:
    is_package_installed('pari')  # optional - build
Expected:
    True
Got:
    False

needs a fix. (I guess dummy packages are not counted as installed).

comment:27 Changed 4 months ago by gh-timokau

  • Cc gh-timokau added

comment:28 Changed 4 months ago by git

  • Commit changed from 72f3c20b38dd0aad6c8325f16f3b4be79b7c70ee to 6eac55053f1e633c4d1638d23070467735149055

Branch pushed to git repo; I updated commit sha1. New commits:

6eac550as pari might become dummy, test with GAP

comment:29 Changed 4 months ago by git

  • Commit changed from 6eac55053f1e633c4d1638d23070467735149055 to ac207d5eed575e96f43ab44475d4a4625639a7a7

Branch pushed to git repo; I updated commit sha1. New commits:

ac207d5don't export GP_DATA_DIR, remove other, unneeded GP* vars

comment:30 Changed 3 months ago by git

  • Commit changed from ac207d5eed575e96f43ab44475d4a4625639a7a7 to 317f0909d0d7ffbd17f32edf99f5b47eae3d6c49

Branch pushed to git repo; I updated commit sha1. New commits:

317f090proper dependency resolution (all tests in PARI spkg-config)

comment:31 Changed 3 months ago by dimpase

  • Status changed from new to needs_review

please review. I only tested this on Debian 10, which has all the require PARI packages. A new Fedora (29 or better) should do, too.

comment:32 Changed 3 months ago by dimpase

The workaeound for docbuilding is not yet in this branch, thus if you supply a multithreaded libpari, docs won't build (unless one uses the workabout in comment:25)

comment:33 Changed 3 months ago by git

  • Commit changed from 317f0909d0d7ffbd17f32edf99f5b47eae3d6c49 to 2c26115cd1752a3728793966785a451b2334a865

Branch pushed to git repo; I updated commit sha1. New commits:

2c26115docbuild workaround for external PARI

comment:34 Changed 3 months ago by dimpase

OK, so it should work with docbuilding too.

By the way, I'd also welcome review of autoconf/m4 code (e.g. m4/sage_check_pari_dep.m4 contains the 1st autoconf macro I ever wrote :-))

comment:35 Changed 3 months ago by isuruf

  • Status changed from needs_review to positive_review

I checked in conda and this picks up the conda installation correctly. For that though I needed to install the optional packages pari-*. conda install pari only pulls in pari-seadata-small and pari-galdata.

Even though the above issue is there, this is still an improvement over current situation.

comment:36 Changed 3 months ago by isuruf

  • Status changed from positive_review to needs_work
configure: === checking whether to install the pari SPKG ===
checking installing gmp/mpir or readline? ... no
checking installing PARI/GP packages? ... checking for gp... /projects/66d93023-00f0-4c12-8a25-5d6d4e486740/sage-build/bin/gp
checking is pari_elldata installed? ... yes
checking is pari_galdata installed? ... yes
checking is pari_galpol installed? ... yes
checking is pari_seadata installed? ...   ***   at top-level: poldegree(ellmodulareqn(211)[1])
  ***                           ^----------------------
  *** ellmodulareqn: error opening seadata file: `/projects/66d93023-00f0-4c12-8a25-5d6d4e486740/sage-build/share/pari/seadata/sea211'.
no; cannot use system pari/GP without seadata package
configure: Install seadata package and reconfigure.
configure: Otherwise Sage will build its own pari/GP.
using Sage's pari SPKG

Above error message from pari should go in config.log and not shown to the user.

comment:37 Changed 3 months ago by git

  • Commit changed from 2c26115cd1752a3728793966785a451b2334a865 to dd2b3faae58e7bad2c3109829b7468eb9902cbf5

Branch pushed to git repo; I updated commit sha1. New commits:

dd2b3faredirect gp errors to config.log

comment:38 Changed 3 months ago by dimpase

  • Status changed from needs_work to needs_review

OK, fixed.

comment:39 Changed 3 months ago by isuruf

  • Reviewers set to Isuru Fernando
  • Status changed from needs_review to positive_review

Thanks. Works for me

comment:40 Changed 3 months ago by fbissey

  • Status changed from positive_review to needs_work

Sorry for the late intervention.

diff --git a/src/sage_setup/docbuild/__init__.py b/src/sage_setup/docbuild/__init__.py
index e406bca..0e07c71 100644
--- a/src/sage_setup/docbuild/__init__.py
+++ b/src/sage_setup/docbuild/__init__.py
@@ -294,7 +294,7 @@ def _build_many(target, args):
             raise
     return ret
 
-if not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3):
+if (os.environ['SAGE_PARI_CFG'] !='') and (not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3)):
     build_many = _build_many
 else:
     # Cygwin 64-bit < 3.0.0 has a bug with exception handling when exceptions

That bit causes me trouble when building the doc

 * python3_7: running distutils-r1_run_phase python_compile_all
Setting permissions of DOT_SAGE directory so only you can read and write it.
Traceback (most recent call last):
  File "sage_setup/docbuild/__main__.py", line 1, in <module>
    from sage_setup.docbuild import main
  File "/dev/shm/portage/sci-mathematics/sage-9999/work/sage-9999/src-python3_7/sage_setup/docbuild/__init__.py", line 297, in <module>
    if (os.environ['SAGE_PARI_CFG'] !='') and (not (CYGWIN_VERSION and CYGWIN_VERSION[0] < 3)):
  File "/usr/lib/python3.7/os.py", line 678, in __getitem__
    raise KeyError(key) from None
KeyError: 'SAGE_PARI_CFG'

comment:41 Changed 3 months ago by fbissey

I would be happy for this ticket to proceed as is and the issue to be addressed in a follow up ticket.

But the general assumption should be sage-on-distros don't run configure. In fact we have pushed a few things so that sage-distro can run without sage-env. I did so a few release ago and Antonio Rojas from arch has pushed so he could follow my lead there in sage-8.9. We can define the variable at build time, I could even flip the value depending on whether pari has been built with tls in Gentoo. But it would be cleaner if it just worked, with the assumption that pari is provided externally, if the variable is undefined.

comment:42 Changed 3 months ago by fbissey

  • Status changed from needs_work to positive_review

Follow up will take place at #28341.

comment:43 Changed 3 months ago by vbraun

  • Branch changed from u/dimpase/packages/pariconfig to dd2b3faae58e7bad2c3109829b7468eb9902cbf5
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:44 follow-up: Changed 3 months ago by dimpase

  • Cc arojas added
  • Commit dd2b3faae58e7bad2c3109829b7468eb9902cbf5 deleted

Can Archlinux get more PARI packages (as much as Debian 10), so that its PARI may be used?

comment:45 in reply to: ↑ 44 Changed 3 months ago by arojas

Replying to dimpase:

Can Archlinux get more PARI packages (as much as Debian 10), so that its PARI may be used?

Standard packages are available from the official repositories, optional packages can easily be installed from AUR - https://aur.archlinux.org/packages/?K=pari-

Note: See TracTickets for help on using tickets.