Make location of the Singular shared library configurable through sage-config

[mkoeppe]

1. For singular, the location of the dynamic library needs to be communicated to the sage runtime.

We add a configuration variable SINGULAR_SO to sage_conf.py.in.

The filename can be found from src/Singular/libSingular.la (currently not installed):

# The name that we can dlopen(3).
dlname='libSingular-4.1.1.dylib'
...
# Directory that this library needs to be installed in:
libdir='/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib'

This will:

• reduce the need for downstream patching (see for example ​https://salsa.debian.org/science-team/sagemath/-/blob/master/debian/patches/d0-singular.patch),

• enable venv builds of sagelib (#30578, #29013) -- for which _get_shared_lib_filename is wrong

• provide preparation for adding an spkg-configure.m4 for singular.

2. src/bin/sage-env unconditionally sets environment variable SINGULARPATH="$SAGE_LOCAL/share/singular"

If SINGULARPATH is already set by the user, perhaps the directory in $SAGE_LOCAL should be prepended instead of overwriting it.

3. src/bin/sage-env unconditionally sets environment variable SINGULAR_EXECUTABLE="$SAGE_LOCAL/bin/Singular".

It should be investigated whether this is actually needed in any supported configuration. If not, remove.

4. src/sage/interfaces/singular.py tries to use SINGULARPATH as if it is a directory name (it is a colon-separated list of directories):

   singular_docdir = SINGULARPATH + "/../info/"
   

This should be fixed so that it works when SINGULARPATH is set by a user to a colon-separated list.

[mkoeppe]

Which versions of singular are good?

[gh-mwageringel]

Replying to mkoeppe:

Which versions of singular are good?

Sage does not currently work correctly with any version of Singular, see #25993.

[dimpase]

well, Debian does have a Singular version which is compatible with their Sage package (v8.6 or so, I guess).

[mkoeppe]

Thanks for the info. I'll check back when #25993 is done.

[gh-thierry-FreeBSD]

This ticket should be listed in #27330 (ATM singular is listed in "No Tickets yet").

[mkoeppe]

Replying to gh-thierry-FreeBSD:

This ticket should be listed in #27330 (ATM singular is listed in "No Tickets yet").

I've added it

[gh-thierry-FreeBSD]

spkg-configure.m4 to be put under /build/pkgs/singular/

[gh-thierry-FreeBSD]

[Copying elements from #29024]

• my 1st problem was a build failure with Singular taken from an external package: to solve it, Singular must be built --with-ntl: it would be fine to add this check to spkg-configure.m4 (but I do not know how!)

• my 2nd error was when building dochtml: NameError: Singular library 'freegb.lib' not found. To solve it, src/sage/env.py and src/bin/sage-env must be modified so that SINGULARPATH is set to singlar's share dir (and not SAGE_SHARE / DESTDIR), and SINGULAR_EXECUTABLE is set to the real Singular and not under the DESTDIR directory.

Now it is OK for me (on FreeBSD).

[dimpase]

the error message

        raise RuntimeError(
            "libSingular not found--a working Singular install in $SAGE_LOCAL "
            "is required for Sage to work")

should be modified, too. Perhaps just remove in $SAGE_LOCAL

[mkoeppe]

Moving this ticket forward is great, but note that it is unrelated to the build problem on homebrew reported on sage-devel... and it is also unlikely that the homebrew version of singular actually works for us (homebrew does not like to patch).

As I explained in ​https://groups.google.com/d/msg/sage-devel/KXK_zxzfhIQ/RO3FryYWAwAJ, on the reporter's system the computed order of -I options is wrong -- something that I was not able reproduce on my system.

[dimpase]

Replying to gh-thierry-FreeBSD:

[Copying elements from #29024]

• my 1st problem was a build failure with Singular taken from an external package: to solve it, Singular must be built --with-ntl: it would be fine to add this check to spkg-configure.m4 (but I do not know how!)

• my 2nd error was when building dochtml: NameError: Singular library 'freegb.lib' not found. To solve it, src/sage/env.py and src/bin/sage-env must be modified so that SINGULARPATH is set to singlar's share dir (and not SAGE_SHARE / DESTDIR), and SINGULAR_EXECUTABLE is set to the real Singular and not under the DESTDIR directory.

Now it is OK for me (on FreeBSD).

The fix in SageMath needs some programming - specifically, the code in singular.pyx calls dlopen on SINGULAR_SO, which is computed by _get_shared_lib_filename dynamically in env.py, and assuming that it is in SAGE_LOCAL. So this probably can instead by handled using cython_aliases from the same module (which does compute all the needed info for Singular, it seems). Or by fixing _get_shared_lib_filename so that it also searches elsewhere.

[mkoeppe]

Is it known whether this comment added in 2009 to singular.pyx is still valid:

    This initializes the SINGULAR library. This is a hack to some
    extent.

    SINGULAR has a concept of compiled extension modules similar to
    Sage. For this, the compiled modules need to see the symbols from
    the main program. However, SINGULAR is a shared library in this
    context these symbols are not known globally. The work around so
    far is to load the library again and to specify ``RTLD_GLOBAL``.

[dimpase]

Someone working on innards of Singular might know. (I'd be surprised if it was changed).

Singular is not alone - this hack is also applied to libgap. I'm inclined to fix _get_shared_lib_filename so that it can accept other libdirs, not only SAGE_LOCAL/lib

[mkoeppe]

Replying to dimpase:

I'm inclined to fix _get_shared_lib_filename so that it can accept other libdirs, not only SAGE_LOCAL/lib

Yes, this function certainly would need changing if we continue to use it. For example, sysconfig.get_config_var('LIBDIR') gives a directory of the system python, not of the current venv.

But I think better solution would be to wrap loading of modules that are linked to singular by ​sys.setdlopenflags.

This is what pytorch does when it has to use RTLD_GLOBAL (see ​https://github.com/pytorch/pytorch/blob/master/torch/__init__.py#L132)

[mkoeppe]

Same at ​https://github.com/tensorflow/tensorflow/blob/9590c4c32dd4346ea5c35673336f5912c6072bf2/tensorflow/python/pywrap_dlopen_global_flags.py and ​https://github.com/tensorflow/tensorflow/blob/9590c4c32dd4346ea5c35673336f5912c6072bf2/tensorflow/python/pywrap_tensorflow.py

[mkoeppe]

I have opened #30561 for this

[mkoeppe]

Hoping we can make progress on this ticket this week - ​https://wiki.sagemath.org/days111

[mkoeppe]

Sage development has entered the release candidate phase for 9.3. Setting a new milestone for this ticket based on a cursory review of ticket status, priority, and last modification date.