Singular does not find "gftables/9" on Solaris

[SimonKing]

The following problem of Singular on Solaris (mark in skynet for sage-4.7.1.rc1, to be precise) was found in #11431, and reported to the ​Singular trac server.

$ ./sage -singular --version
   ? cannot open `help.cnf`
Singular for SunOS-5 version 3-1-1 (3114-2011080223)  Aug  2 2011 23:47:38
with
        factory(@(#) factoryVersion = 3.1.1),libfac(3.1.1,Feb 2010),
        GMP(4.2),NTL(5.5.2),32bit,static readline,Plural,DBM,
        dynamic modules,OM_NDEBUG,random=1312397091
        CC= gcc -O2 -g -fPIC -pipe -DNDEBUG -DOM_NDEBUG -DSunOS_5 -DHAVE_CONFIG_H,
        CXX= g++ -O2 -g -fPIC -pipe -DNDEBUG -DOM_NDEBUG -DSunOS_5 -DHAVE_CONFIG_H (4.5.1)
argv[0]   :     Singular-3-1-1
SearchPath:     /home/simonking/SAGE/sage-4.7.1.rc1mark/local/share/singular:/home/simonking/SAGE/sage-4.7.1.rc1mark/local/LIB
Singular  :     /home/simonking/SAGE/sage-4.7.1.rc1mark/local/bin/Singular
BinDir    :     /home/simonking/SAGE/sage-4.7.1.rc1mark/local/bin
RootDir   :     /home/simonking/SAGE/sage-4.7.1.rc1mark/local
DefaultDir:     /home/simonking/SAGE/sage-4.7.1.rc1mark/local
InfoFile  :
IdxFile   :
HtmlDir   :
ManualUrl :     http://www.singular.uni-kl.de/Manual/3-1-1
ExDir     :
Path      :     /home/simonking/SAGE/sage-4.7.1.rc1mark/local/bin:/home/simonking/SAGE/sage-4.7.1.rc1mark:/usr/local/gcc-4.5.1/sparc-SunOS-ultrasparc3-sun-as-ld/bin:/usr/local/bin/sparc-SunOS-ultrasparc3:/usr/bin:/usr/ccs/bin
EmacsDir  :
Available HelpBrowsers: dummy, emacs,
Current HelpBrowser: dummy
                     SINGULAR                             /  Development
 A Computer Algebra System for Polynomial Computations   /   version 3-1-1
                                                       0<
     by: G.-M. Greuel, G. Pfister, H. Schoenemann        \   Feb 2010
FB Mathematik der Universitaet, D-67653 Kaiserslautern    \
> ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp);
   ? cannot open `gftables/9`
   ? cannot make ring
   ? error occurred in or before STDIN line 1: `ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp);`
   ? expected ring-expression. type 'help ring;'
>

That example works fine on most platforms; according to the buildbot, t2, mark and mark2 are the only exceptions.

New spkg: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-1-4.p12.spkg

[SimonKing]

Oleksander and Hannes from the Singular team replied. Apparently, there should be tables in the folder SAGE_ROOT/local/share/singular/gftables (or perhaps in SAGE_ROOT/local/LIB/gftables).

In my Sage installation on mark, there is a gftables directory, but it only contains a table for the field of size 10201.

I am not sure whether the tables should already be created when Singular is built, or only when a field of a particular size is first created (I asked the Singular team a few minues ago).

According to the Singular manual, Singular allows all field sizes below 2^16. Looking at the gftables directory of my installation on my linux computer, clearly there are less tables than field sizes. So, I guess that the tables are only created when needed.

But why is Singular not able to? It seems that SINGULARPATH is available:

sage: [(s,v) for s,v in os.environ.iteritems() if 'SINGULAR' in s]
[('SINGULAR_EXECUTABLE', '/home/simonking/SAGE/sage-4.7.1.rc1mark/local/bin/Singular'), ('SINGULARPATH', '/home/simonking/SAGE/sage-4.7.1.rc1mark/local/share/singular')]

[SimonKing]

Replying to SimonKing:

According to the Singular manual, Singular allows all field sizes below 2^16. Looking at the gftables directory of my installation on my linux computer, clearly there are less tables than field sizes. So, I guess that the tables are only created when needed.

I was wrong: The gftables file contains files for all prime powers greater than 1 and smaller than 2^16 that are not prime. So, that looks as if it was made at compilation time.

[SimonKing]

The following indicates that the problem is on Sage's end:

sage: singular.eval('ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp)')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)

/home/simonking/SAGE/sage-4.7.1.rc1mark/<ipython console> in <module>()

/home/simonking/SAGE/sage-4.7.1.rc1mark/local/lib/python2.6/site-packages/sage/interfaces/singular.pyc in eval(self, x, allow_semicolon, strip, **kwds)
    548
    549         if s.find("error") != -1 or s.find("Segment fault") != -1:
--> 550             raise RuntimeError, 'Singular error:\n%s'%s
    551
    552         if get_verbose() > 0:

RuntimeError: Singular error:
   ? cannot open `gftables/9`
   ? cannot make ring
   ? error occurred in or before STDIN line 39: `ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp);`
   ? expected ring-expression. type 'help ring;'
sage: R.<a,b,c,d,e,f> = GF(9,'x')[]
sage: singular(R)
//   characteristic : 3
//   1 parameter    : x
//   minpoly        : (x^2-x-1)
//   number of vars : 6
//        block   1 : ordering dp
//                  : names    a b c d e f
//        block   2 : ordering C
sage: singular.eval('ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp)')
'ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp);'

In other words: If one constructs a polynomial ring over a finite non-prime field and converts it into Singular, then the missing file in the gftables-folder is created!

In other words: If nothing else helps then one could construct while Sage is built a polynomial ring over any finite non-prime field of size less than 2^16 and convert it into Singular. In that way, all gftables would be created.

That still does not explain why the field tables are not created (or copied?) in the first place, but it could be a work around.

[SimonKing]

Hi Martin,

perhaps you know about missing "gftables" in Singular.

For working with finite non-prime fields, Singular needs certain files in a folder gftables. However, these files are missing when Sage is built on Solaris.

Problem: When Singular tries to create a polynomial ring over a finite field and the table for that field is missing, then it raises an error, but it does not simply compute that table -- at least not on Solaris.

However, when one has a polynomial ring over the same finite field in Sage (using libsingular) and converts it into Singular then the field table magically appears.

Questions to Martin: Did you do anything special in the Singular conversion method of libsingular polynomial rings? That's to say, do you catch errors mentioning gftables? Is there a command to create a gftable?

[SimonKing]

Actually, it is even better: The missing table is not only created when a libsingular ring is converted into Singular, but already when that ring is created. Proof:

I emptied the gftables folder and started Sage. Then:

sage: singular.eval('ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp)')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
...
RuntimeError: Singular error:
   ? cannot open `gftables/9`
   ? cannot make ring
   ? error occurred in or before STDIN line 25: `ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp);`

Hence, Singular does not create the table by itself.

However, creating a ring in libsingular means that the table shows up and can be used:

sage: R.<a,b,c,d,e,f> = GF(9,'x')[]
sage: singular.eval('ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp)')
'ring r1 = (9,x),(a,b,c,d,e,f),(M((1,2,3,0)),wp(2,3),lp);'

So, question to Martin: How does libsingular make sure that a field table is present?

[SimonKing]

Now I am really puzzled. On Singular trac, Hannes stated: "No, gftables are parts of the sources and will be copied during installation, but usually not computed (the program for that, a part of factory, will not be build during a standard compilation and installation)."

That gives rise to two questions:

1. Why are the tables not copied to the appropriate directory on Solaris?

2. How can libsingular create the missing tables?

Any clue?

I am also modifying the "reported upstream" part of the ticket. I think that meanwhile it is clear that the problem is not an upstream problem.

[jhpalmieri]

From the singular installation log on mark2:

./mkinstalldirs /home/palmieri/mark2/sage-4.7.1.rc1/local/share/singular/gftables
mkdir /home/palmieri/mark2/sage-4.7.1.rc1/local/share/singular/gftables
./install-sh -c -m 644 `pwd`/LIB/gftables/* /home/palmieri/mark2/sage-4.7.1.rc1/local/share/singular/gftables/
make[3]: Leaving directory `/home/palmieri/mark2/sage-4.7.1.rc1/spkg/build/singular-3-1-1-4.p9/src/Singular'

I wonder if the script install-sh is the problem.

[leif]

Off-list ;-) comment:

$ find /home/leif/Sage/spkgs/singular-3-1-1-4.p9/ -name gftables -print -exec ls {} \;
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/Singular/LIB/gftables
10201  125    16     1849   2209   25     289    32768  38809  4913   529    625    7921  961
1024   12769  16129  18769  22201  256    29791  343    39601  49729  5329   63001  8
10609  128    16384  19321  22801  26569  29929  3481   4      5041   54289  64     81
11449  1331   16807  19683  2401   27     3125   361    4096   50653  57121  6561   8192
11881  1369   1681   2048   243    27889  32     36481  44521  512    58081  6859   841
121    14641  169    2187   24389  2809   32041  3721   4489   51529  59049  6889   9
12167  15625  17161  2197   24649  28561  32761  37249  49     52441  6241   729    9409

I.e., they're all (precomputed) in our upstream tree, and should get copied by some BSD install program, cf. my reply on sage-release.

Usually autotools-enabled packages ship with some default / fallback install shell script in case the system lacks one, and so does Singular:

$ find /home/leif/Sage/spkgs/singular-3-1-1-4.p9/ -name install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/libfac/install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/Singular/install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/kernel/install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/factory/bin/install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/src/omalloc/install-sh
/home/leif/Sage/spkgs/singular-3-1-1-4.p9/install-sh

spkg/logs/singular-*.log should show which install actually got used:

checking for a BSD compatible install... /usr/bin/install -c

(on my Linux box)

So maybe the one found on Solaris / mark etc. is broken, but configure doesn't notice that. It might just install the first file passed to it, which would explain why just 10201 (the first in lexicographical order) is present there, as the relevant command used by Singular is

/usr/bin/install -c -m 644 `pwd`/LIB/gftables/* <SAGE_ROOT>/local/share/singular/gftables/

(Notice the wildcard, which is expanded by the shell.)

Furthermore, gftables/* is the only such instance; all other files are installed by something like

for f in whatever/*; do
    install -c [...] $f
done

so IMHO this is an upstream problem, since they should either use the latter for gftables/* as well, or make sure the install program they use supports multiple files at once.

[leif]

Replying to jhpalmieri:

From the singular installation log on mark2:

./install-sh -c -m 644 `pwd`/LIB/gftables/* /home/palmieri/mark2/...

I wonder if the script install-sh is the problem.

Haha, their "own"! XD

[leif]

For the record, I've re-reported this upstream (the installation bug).

[SimonKing]

Replying to leif:

For the record, I've re-reported this upstream (the installation bug).

... actually together with a patch!

If Singular accepts the patch, then it will (hopefully) be part of the next Singular release. When? Singular releases take place usually every 6 months. The last was in April. So, it will hopefully be in October.

Then, the upstream fix will be included in Sage. When? Currently, Sage has Singular-3-1-1, which was released in March, 2010. So, the upstream fix will be in Sage, optimistically, in early 2012.

Hence, I'd appreciate if we could find a temporary solution in Sage. Two approaches:

1. Since libsingular is able to create the missing files, we could append a section to the Sage install script that creates a libsingular ring over any finite non-prime field of order less than 2^16.

2. We could patch the current Singular spkg.

Probably, the second solution is better.

[leif]

Replying to SimonKing:

Replying to leif:

For the record, I've re-reported this upstream (the installation bug).

... actually together with a patch!

Yes, but had problems adding a further comment on it; looked as if they blocked me... :D (different browsers from the same machine)

If Singular accepts the patch, then it will (hopefully) be part of the next Singular release. When? Singular releases take place usually every 6 months. The last was in April. So, it will hopefully be in October.

Then, the upstream fix will be included in Sage. When? Currently, Sage has Singular-3-1-1, which was released in March, 2010. So, the upstream fix will be in Sage, optimistically, in early 2012.

Haven't yet looked for other (Sage) Singular tickets, so we might already have some dealing with our current version, or even one which upgrades to a newer upstream release.

2. We could patch the current Singular spkg.

In any case, we can of course apply the same patch in the next Singular spkg.

[leif]

Replying to leif:

Haven't yet looked for other (Sage) Singular tickets, so we might already have some dealing with our current version, or even one which upgrades to a newer upstream release.

So far I've found #10903 (new, for upgrading Singular) and a 3-1-1-4.p10 spkg (#11550), already merged into the inofficial / not yet announced Sage 4.7.2.alpha0 (among about a dozen other open tickets related to Singular).

2. We could patch the current Singular spkg.

In any case, we can of course apply the same patch in the next Singular spkg.

So I could provide a p11 fixing this based on that p10.

I don't know if you plan to make further changes (to the Sage library I guess) on this ticket; if so, I can open a new ticket for the spkg. Just let me know.

P.S.: What's up with the apparently missing help.cnf? (Cf. #11519)

[leif]

Hannes confirmed it's a Singular bug, "fixed in revision 14360".

I don't know which Singular version this revision corresponds to though.

[SimonKing]

Replying to leif:

Hannes confirmed it's a Singular bug, "fixed in revision 14360".

I don't know which Singular version this revision corresponds to though.

It probably doesn't matter which: The version in Sage is strongly outdated. Singular 3-1-3 is out since April, but Sage is still using one year old Singular 3-1-1.

Anyway. My guess is that the fix will be part of Singular-3-1-4. In any case, we probably ought to take care of it NOW.

[leif]

Plain patch to copy into Singular's spkg (patches/) and apply from within spkg-install.

[leif]

I've just attached a copy of the patch I submitted upstream.

Note that we already patch src/Singular/Makefile.in (with patches/Singular-Makefile.in.patch), so the two patches should be merged into one.

(We currently copy a patched file anyway.)

[leif]

Replaces patches/Singular-Makefile.in in the Singular spkg. (Cumulatively patched src/Singular/Makefile.in.)

[leif]

Replaces patches/Singular-Makefile.in.patch in the Singular spkg. (Cumulative patch to src/Singular/Makefile.in.)

[leif]

Diff between Jeroen's p11 Singular spkg from #11663 and what we would get here (with the two files replaced). For reference / review. There's no Changelog entry yet.

[jdemeyer]

Just wondering: is there any reason for

for file in `pwd`/LIB/gftables/*; do

instead of simply

for file in LIB/gftables/*; do

[jdemeyer]

This will be folded into #11663.

[jdemeyer]

New spkg based on the one for #11663: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-1-4.p12.spkg, needs review.

[jdemeyer]

Diff for the Singular spkg, for reviewers

[leif]

Replying to jdemeyer:

Just wondering: is there any reason for

for file in `pwd`/LIB/gftables/*; do

instead of simply

for file in LIB/gftables/*; do

No, that's just upstream style I didn't change because I submitted the (same) patch there in the first place. ;-)

Seriously, the first one is safer since pwd yields an absolute path.

[leif]

Replying to jdemeyer:

New spkg based on the one for #11663: ​http://boxen.math.washington.edu/home/jdemeyer/spkg/singular-3-1-1-4.p12.spkg, needs review.

I don't know why you didn't just take the replacement files I attached, but for the record: It's not an upstream patch, but a patch I submitted upstream, i.e. I'm the author. ;-)

If you merge this (and #11663) into Sage 4.7.1, I think we should also fix the trivial missing help.cnf which we have to also "manually" copy with cp -p in spkg-install's install_docs(); just add

    cp -p "$SRC/Singular/LIB/help.cnf" "$SAGE_LOCAL/share/singular/"
    if [ $? -ne 0 ]; then
        echo "Error copying help configuration file"
        exit 1
    fi

there, or changed it by

spkg-install

@@ -356 +356 @@
 
 install_docs()
 {
-    cp -p "$SHARED/singular.hlp" "$SHARED/singular.idx" "$SAGE_LOCAL/share/singular/"
+    cp -p "$SHARED/singular.hlp" "$SHARED/singular.idx" "$SRC/Singular/LIB/help.cnf" \
+          "$SAGE_LOCAL/share/singular/"
     if [ $? -ne 0 ]; then
         echo "Error installing documentation"
         exit 1

which fixes #11519 and (part of) #5994.

[leif]

Patch to Singular's spkg-install installing the otherwise missing help.cnf. (This actually fixes #11519, and partially #5994.)

[leif]

Replying to leif:

... or change it by

spkg-install

@@ -356 +356 @@
 
 install_docs()
 {
-    cp -p "$SHARED/singular.hlp" "$SHARED/singular.idx" "$SAGE_LOCAL/share/singular/"
+    cp -p "$SHARED/singular.hlp" "$SHARED/singular.idx" "$SRC/Singular/LIB/help.cnf" \
+          "$SAGE_LOCAL/share/singular/"
     if [ $? -ne 0 ]; then
         echo "Error installing documentation"
         exit 1

which fixes #11519 and (part of) #5994.

For convenience, since we still don't get trac notifications, I've now also attached a patch for that here: trac_11645-fix_missing_help.cnf_file.patch​

[kcrisman]

It would be really kind to make the help.cnf thing a different ticket. I simply don't have the time to keep rebasing to this ticket with #11550; it is not an easy process for me.

[leif]

Replying to kcrisman:

It would be really kind to make the help.cnf thing a different ticket. I simply don't have the time to keep rebasing to this ticket with #11550; it is not an easy process for me.

Well, you shouldn't rebase too early. We meanwhile have enough chaos anyway, multiplied by the fact that we still don't get trac notifications.

I gave #11663 positive review to be able to quickly base my new Singular spkg on yours after you had rebased that on Jeroen's. I asked multiple times where we can fix what or in which order.

It doesn't make sense to me to include (merge) one trivial patch, but not the other one. Permissions can btw. also trivially be fixed by the one who provides an installation, while it is harder to afterwards install some files located somewhere inside spkgs.

[jdemeyer]

Replying to kcrisman:

It would be really kind to make the help.cnf thing a different ticket.

I totally agree. We cannot just keep adding new stuff here all the time. This ticket is about the much more serious issue of gftables. You are talking about "trivial patches" but I think a trivial patch for a serious issue is very different from a trivial patch for a trivial issue.

I do agree with leif about the rebasing: simply wait until the dust settles over this ticket (i.e. when I gets merged), then rebase.

I have changed the spkg just to change the wording in SPKG.txt to:

=== singular-3-1-1-4.p12 (Leif Leonhardy, Jeroen Demeyer, 10 August 2011) ===
 * Trac #11645: Fix installation of gftables on Solaris
   by applying a patch to Singular/Makefile.in

I also add a doctest. Both the spkg and doctest need review.

[jdemeyer]

Doctest to be added

[SimonKing]

Question: What is needed to do in order to test? I suppose I have to install the new spkg (on Solaris, preferably), best after deleting gftables (just to be on the safe side), and then? Should I use trac_11645-fix_missing_help.cnf_file.patch as well?

[SimonKing]

Replying to SimonKing:

Question: What is needed to do in order to test? I suppose I have to install the new spkg (on Solaris, preferably), best after deleting gftables (just to be on the safe side), and then? Should I use trac_11645-fix_missing_help.cnf_file.patch as well?

Or only 11645_doctest.patch after the spkg?

[SimonKing]

Replying to SimonKing:

Should I use trac_11645-fix_missing_help.cnf_file.patch as well?

Probably not: Apparently that patch is part of the spkg, right?

So, it only is the doctest patch, isn't it?

[jdemeyer]

Replying to SimonKing:

Question: What is needed to do in order to test? I suppose I have to install the new spkg (on Solaris, preferably), best after deleting gftables (just to be on the safe side), and then?

I guess you should at least doctest sage/interfaces/singular.py including my newly added doctest. This should prove that the gftables was installed. To be sure, you can then delete (or chmod 0000) gftables and check that the doctest does indeed fail.

Should I use trac_11645-fix_missing_help.cnf_file.patch as well?

No. This is a different issue.

[leif]

Replying to jdemeyer:

I have changed the spkg just to change the wording in SPKG.txt

The problem is rather unrelated to Solaris, it's just that the machines happened to have no BSD install installed such that one of those (broken) install scripts shipped with Singular got used.

I'd also add to Special Update/Build? Instructions that this is said to be fixed in later stable releases (but not necessarily the next one we'll upgrade to).

(Currently the patches aren't documented well; the whole spkg needs work on some follow-up.)

[leif]

Replying to SimonKing:

Question: What is needed to do in order to test? I suppose I have to install the new spkg (on Solaris, preferably)

You can test the spkg on any machine where command -v install doesn't show anything, or by simply renaming (or chmod a-x) the installed one, usually /usr/bin/install.

best after deleting gftables (just to be on the safe side)

Yes, of course.

[SimonKing]

This is what I did to test, on the skynet machine mark (which is solaris and which did expose the bug). And, concerning Leif's hint:

$ command -v install
-bash: command: install: not found

1. I removed the contents of SAGE_ROOT/local/share/singular/gftables

2. I installed ​the new spkg. Now, gftables is filled with all files that are expected. So, that problem seems solved.

3. I applied 11645_doctest.patch​ and did sage -br. Then, I tried singular.eval('ring r = (25,x),(a,b,c,d,e,f),dp'), which worked fine and would have failed without gftables.

4. sage -t sage/interfaces/singular.py worked fine.

5. I opened the spkg, verified that the changelog is present in SPKG.txt and that hg status does not complain. It was all fine.

Since the Singular code did not change (only the gftables files were copied), I think it is enough for a positive review.

Please verify whether I correctly filled the "Author(s)" field!

Caveat

I am currently running sage -t sage/rings/, which takes painfully long on mark. The tests of the first 25 files passed, but of course if some test fails in the next few hours then I'll complain.

[SimonKing]

FWIW, the tests in sage/rings have all worked well on mark. So, no complaint from my side, there remains the positive review.