Opened 6 years ago

Closed 6 years ago

#14909 closed defect (fixed)

Gap package HAP does not load

Reported by: vbraun Owned by:
Priority: major Milestone: sage-5.12
Component: packages: optional Keywords:
Cc: Snark, mmarco, tscrim, dimpase, niles, wdj Merged in: sage-5.12.beta0
Authors: Volker Braun Reviewers: Niles Johnson
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by vbraun)

The optional gap_packages spkg contains HAP, but it fails to load because of missing dependency on polycyclic. See http://ask.sagemath.org/question/2822/how-to-load-optional-gap-packages

Updated optional spkg: http://boxen.math.washington.edu/home/vbraun/spkg/gap_packages-4.6.4.p1.spkg

Apply trac_14909_gap_packages_doctests.patch to the Sage library

Attachments (1)

trac_14909_gap_packages_doctests.patch (11.1 KB) - added by vbraun 6 years ago.
Updated patch

Download all attachments as: .zip

Change History (17)

comment:1 Changed 6 years ago by niles

  • Cc Snark mmarco tscrim dimpase niles added

It seems that a number of packages fail to load, possibly for similar reasons. Perhaps the people from #14682 know how to fix this:

sage: optional_packages()[0]
['database_gap-4.6.4', 'gap_packages-4.6.4']
sage: version()
'Sage Version 5.11.beta3, Release Date: 2013-06-21'
sage: gap_package_names = ['braid','crime','cryst','ctbllib','design','factint','gapdoc','grape','guava','hap','hapcryst','happrime','laguna','polymake','sonata','toric']

sage: for name in gap_package_names:
    stat = gap('LoadPackage("{0}")'.format(name))
    prefix = ' ' if str(stat) == 'true' else 'x'
    print("{2} : {0: <8} got '{1}' when loading.".format(name,stat,prefix))
....:     

  : braid    got 'true' when loading.
  : crime    got 'true' when loading.
x : cryst    got 'fail' when loading.
  : ctbllib  got 'true' when loading.
  : design   got 'true' when loading.
  : factint  got 'true' when loading.
  : gapdoc   got 'true' when loading.
  : grape    got 'true' when loading.
  : guava    got 'true' when loading.
x : hap      got 'fail' when loading.
x : hapcryst got 'fail' when loading.
x : happrime got 'fail' when loading.
  : laguna   got 'true' when loading.
x : polymake got 'fail' when loading.
  : sonata   got 'true' when loading.
  : toric    got 'true' when loading.

This leads to a number of failing doctests:

$ ./sage -t --optional=gap_packages devel/sage/sage

----------------------------------------------------------------------
sage -t devel/sage/sage/coding/linear_code.py  # 12 doctests failed
sage -t devel/sage/sage/combinat/designs/incidence_structures.py  # 2 doctests failed
sage -t devel/sage/sage/combinat/designs/design_catalog.py  # 1 doctest failed
sage -t devel/sage/sage/groups/perm_gps/permgroup.py  # 19 doctests failed
----------------------------------------------------------------------

comment:2 Changed 6 years ago by vbraun

Fixed by adding missing dependency packages. Will upload spkg tomorrow.

Also, HAPprime was never included, fixed in the docs.

The packages cryst isn't included and therefore HAPcryst can't work. I'll leave this for later.

comment:3 Changed 6 years ago by vbraun

  • Authors set to Volker Braun
  • Description modified (diff)
  • Status changed from new to needs_review

comment:4 Changed 6 years ago by niles

  • Branch set to u/niles/14909
  • Status changed from needs_review to needs_info

I had trouble with this at first, and then I realized one needs to use --optional=sage,gap_packages; without sage there, _only_ the #optional lines are run, which makes a lot of tests fail. (The tests above still fail with --optional=sage,gap_packages though.)

I like the idea of adding tests for the gap packages, but maybe the test in this patch is too verbose. It would fail, for example, if the user had some extra gap packages installed manually, even if they load correctly. Thoughts on changing it to something like the following?

    sage: from sage.tests.gap_packages import all_installed_packages                  
    sage: for name in all_installed_packages():       # optional: gap_packages        
    ....:     stat = libgap.eval('LoadPackage("{0}")'.format(name))                   
    ....:     message = "{2} : {0: <10} got '{1}' when loading."                      
    ....:     if str(stat) == 'true':                                                 
    ....:         prefix = ' '                                                        
    ....:     else:                                                                   
    ....:         prefix = 'x'                                                        
    ....:         print(message.format(name,stat,prefix))                                                                                 
    x : HAPcryst   got 'fail' when loading.       

In fact, maybe I have attached a git branch with this version of the test! (Still figuring out the git workflow . . .)

And another issue: Why do we have

$ ./sage -t --optional=sage,gap_packages src/sage/tests/gap_packages.py 
Running doctests with ID 2013-07-19-15-20-57-fee4b5ac.
...
Total time for all tests: 7.3 seconds
    cpu time: 7.2 seconds
    cumulative wall time: 7.2 seconds

and

sage: %time import sage.tests.gap_packages
CPU times: user 5.55 s, sys: 0.05 s, total: 5.61 s
Wall time: 5.61 s

The rest of the things in sage/tests seem to import instantly . . . is there something I'm doing wrong which makes this one so slow?

comment:5 Changed 6 years ago by niles

  • Branch u/niles/14909 deleted

Actually, maybe I should leave the "Branch" field for the branch which the patch author might submit, so I can be the patch reviewer. I guess someone who finds it useful can view my commits at

http://trac.sagemath.org/browser/?rev=6b5515c48c7440a7c15e81937883a39130bbff8e

and (somehow) merge the branch u/niles/14909.

comment:6 Changed 6 years ago by vbraun

  • Status changed from needs_info to needs_review

I've updated the patch to only check for the failures and some misc cleanup. Since we don't have a way of working with optional spkgs yet I propose to stick to the old workflow in this ticket.

Startup is probably slow since this makes use of libgap, which doesn't yet use cached workspaces (search trac if you want to know more)

comment:7 Changed 6 years ago by niles

  • Status changed from needs_review to needs_info

Indeed, I have verified that all of the import time comes from importing libgap. The new version of the patch looks good, and in particular the new test for loading packages is much better.

Now checking (long) doctests, I find failures in 3 files:

----------------------------------------------------------------------
sage -t --long sage/combinat/designs/incidence_structures.py  # 2 doctests failed
sage -t --long sage/combinat/designs/design_catalog.py  # 1 doctest failed
sage -t --long sage/coding/linear_code.py  # 8 doctests failed
----------------------------------------------------------------------

The first two seem to be introduced by #14499; fixing these is now #14950

The failures in coding/linear_code are all in optional gap_packages doctests, and they all have to do with the GUAVA package. My guess is that these come from a version change or something similar, because guava does load, and some guava-dependent doctests do work, such as

sage: C = HammingCode(5,GF(2))
sage: C.covering_radius()  # optional - gap_packages (Guava package)
1
sage: C = HammingCode(3,GF(2))
sage: MS = MatrixSpace(GF(2),1,7)
sage: F = GF(2); a = F.gen()
sage: v1 = [a,a,F(0),a,a,F(0),a]
sage: C.decode(v1,algorithm="guava")  # optional - gap_packages (Guava package)

One failure comes from

sage: print bounds_minimum_distance(10,5,GF(2)) # optional - gap_packages (Guava package)

This seems to give the same record as expected (see here), but the entries aren't printed in the correct order, hence causing a doctest failure.

Another failure comes from

sage: MS = MatrixSpace(GF(2),4,7)
sage: G = MS([[1,1,1,0,0,0,0],[1,0,0,1,1,0,0],[0,1,0,1,0,1,0],[1,1,0,1,0,0,1]])
sage: C = LinearCode(G)
sage: C.spectrum()
[1, 0, 0, 7, 7, 0, 0, 1]
sage: C.spectrum(algorithm="leon")     # optional - gap_packages (Guava package)
sh: /home/johnson.5320/sage/local/gap/latest/pkg/guava-3.12/bin/wtdist: No such file or directory
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
...
RuntimeError: Problem calling Leon's wtdist program. Install gap_packages*.spkg and run './configure ../..; make'.

For this last error, I see that the file

SAGE_ROOT/local/gap/latest/pkg/guava-3.12/bin/wtdist

does not exist, but

SAGE_ROOT/local/gap/latest/pkg/guava-3.12/bin/x86_64-unknown-linux-gnu-gcc-default64/wtdist

does exist. So probably this is an error coming from rearranging files. I haven't checked the other failures here; I am inclined to believe that fixing them belongs on a new ticket, but I'm not sure.

Changed 6 years ago by vbraun

Updated patch

comment:8 follow-up: Changed 6 years ago by vbraun

I've fixed the guava package path and the trivial failures. There are still three failures in linear_code.py with guava that seem to be bugs in guava. For all of them we have native code that does the same, only the optional guava-based implementation fails. Needs to be debugged further by somebody else.

comment:9 in reply to: ↑ 8 Changed 6 years ago by dimpase

Replying to vbraun:

I've fixed the guava package path and the trivial failures. There are still three failures in linear_code.py with guava that seem to be bugs in guava. For all of them we have native code that does the same, only the optional guava-based implementation fails. Needs to be debugged further by somebody else.

is this a libgap-related issue? I'm not aware of guava bugs in the "main" Sage code.

comment:10 Changed 6 years ago by niles

It could be a libgap issue; I've pasted the output of the 3 failing tests below. Since we are now talking about nontrivial problems with either guava or libgap on a ticket whose issue was loading the HAP package, I propose that we finish this one and open a new ticket for the guava failures.

While we're at it, we might try to think of ways to ensure that the optional doctests are somehow run more regularly. I don't see how to phrase this as a reasonable ticket (it seems more like a general complaint than a specific fixable problem), and I imagine there have already been some attempts at doing this.

File "sage/coding/linear_code.py", line 1311, in sage.coding.linear_code.LinearCode.decode
Failed example:
    C.decode(v, algorithm="guava")  # optional - gap_packages (Guava package)
Exception raised:
    Traceback (most recent call last):
    ...
    RuntimeError: Gap produced error output
    Error, List Element: <list>[2] must have an assigned value

       executing Decodeword($sage11,$sage32);
File "sage/coding/linear_code.py", line 2165, in sage.coding.linear_code.LinearCode.permutation_automorphism_group
Failed example:
    C.permutation_automorphism_group(algorithm="gap")  # optional - gap_packages (Guava package)
Exception raised:
    Traceback (most recent call last):
    ...
    TypeError: Gap produced error output
    Error, no method found! For debugging hints type ?Recovery from NoMethodFound
    Error, no 1st choice method found for `MatrixAutomorphisms' on 1 arguments

       executing $sage19:=MatrixAutomorphisms(matCwt);;
File "sage/coding/linear_code.py", line 2167, in sage.coding.linear_code.LinearCode.permutation_automorphism_group
Failed example:
    C.permutation_automorphism_group(algorithm="gap")  # optional - gap_packages (Guava package)
Exception raised:
    Traceback (most recent call last):
    ...
    TypeError: Gap produced error output
    Error, no method found! For debugging hints type ?Recovery from NoMethodFound
    Error, no 1st choice method found for `MatrixAutomorphisms' on 1 arguments

       executing $sage31:=MatrixAutomorphisms(matCwt);;

comment:11 follow-up: Changed 6 years ago by vbraun

None of the failing doctests uses libgap, its all ancient code using the gap pexpect interface. At least one of them works fine with prime finite fields but fails in the non-prime case:

            sage: F.<a> = GF(4)
            sage: C = HammingCode(2,F)
            sage: v = vector(F, [1,0,0,a,1])
            sage: C.decode(v)
            (a + 1, 0, 0, a, 1)
            sage: C.decode(v, algorithm="nearest neighbor")
            (a + 1, 0, 0, a, 1)
            sage: C.decode(v, algorithm="guava")  # optional - gap_packages (Guava package)
            BOOM!

comment:12 Changed 6 years ago by vbraun

  • Cc wdj added
  • Status changed from needs_info to needs_review

comment:13 in reply to: ↑ 11 Changed 6 years ago by niles

Replying to vbraun:

None of the failing doctests uses libgap, its all ancient code using the gap pexpect interface.

Seems like an even stronger reason to move this to a new ticket. I'll do that tomorrow unless someone objects.

comment:14 Changed 6 years ago by niles

  • Reviewers set to Niles Johnson
  • Status changed from needs_review to positive_review

The remaining problems in linear_code.py have been reported at #14979. The patch here resolves the issue in this ticket, and more!

comment:15 Changed 6 years ago by schilly

updated spkg landed on the servers …

comment:16 Changed 6 years ago by jdemeyer

  • Merged in set to sage-5.12.beta0
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.