Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#13588 closed enhancement (fixed)

libGAP! -- create a Cython library interface to gap

Reported by: vbraun Owned by: joyner
Priority: major Milestone: sage-5.7
Component: group theory Keywords:
Cc: mmarco, tfeulner, iandrus, SimonKing Merged in: sage-5.7.beta1
Authors: Volker Braun Reviewers: Dmitrii Pasechnik
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: #13211, #13880 Stopgaps:

Status badges

Description (last modified by jdemeyer)

Currently, libGAP doesn't give useful errors if something goes wrong during initialization.

Also, the GAP SIGINT handler got installed, this is fixed in the new spkg. Now Ctrl-C works as expected.

The trac_6391_* patch has already been reviewed at #6391.

Attachments (8)

trac_13588_improve_libGAP.patch (16.6 KB) - added by vbraun 9 years ago.
Updated patch
trac_6391_libGAP.patch (140.6 KB) - added by vbraun 9 years ago.
patch from #6391
trac_6391_libGAP_root.patch (1.7 KB) - added by vbraun 9 years ago.
patch from #6391
trac_13588_exec_fix.patch (750 bytes) - added by vbraun 9 years ago.
Updated patch
trac_13588_manifest.patch (538 bytes) - added by vbraun 9 years ago.
Initial patch
13588_libGAP_root.patch (2.4 KB) - added by jdemeyer 9 years ago.
libgap_sparc.patch (1.2 KB) - added by jdemeyer 9 years ago.
Patch needed for SPARC
trac_13588_docbuild.patch (2.9 KB) - added by vbraun 9 years ago.
Updated patch

Download all attachments as: .zip

Change History (81)

comment:1 Changed 9 years ago by vbraun

  • Authors set to Volker Braun
  • Cc mmarco added
  • Dependencies set to #6391, #13211
  • Status changed from new to needs_review

Miguel, you said that you had some mysterious error. Can you try this patch? It depends on the recently-updated patches on #6391, #13211.

comment:2 Changed 9 years ago by mmarco

I have been checking the different updates of libgap, and the error seemed to go away several versions ago. Anyways, i will keep testing it.

comment:3 Changed 9 years ago by mmarco

I don't know if it is an issue related with this patch or if it comes from libgap itself, but i have experienced some problems when i use tab completion in a session in the command line (i haven't checked it in the notebook).

An example:

sage: F=libgap.eval('FreeGroup(2)')
sage: (a,b)=F.GeneratorsOfGroup()
sage: H=F/libgap([a^2,b^2,a*b*a*b])
sage: H
<fp group on the generators [ f1, f2 ]>
sage: H.IsomorphismSimplifiedFpGroup()
[ f1, f2 ] -> [ f1, f2 ]
sage: H=F/libgap([a^2,b^2,a*b])    
sage: c=H.Gene
H.GeneralLinearGroup      H.GeneralisedEigenspaces  H.GeneralizedEigenvalues  H.GeneratorsOfIdeal
H.GeneralOrthogonalGroup  H.GeneralisedEigenvalues  H.GeneratorsOfField       
H.GeneralUnitaryGroup     H.GeneralizedEigenspaces  H.GeneratorsOfGroup       
sage: c=H.GeneratorsOfGroup()[0]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/mmarco/sage-5.4/<ipython console> in <module>()

/home/mmarco/sage-5.4/local/lib/python2.7/site-packages/sage/libs/gap/element.so in sage.libs.gap.element.GapElement.__getattr__ (sage/libs/gap/element.c:3555)()

AttributeError: Name "GeneratorsOfGroup" is not defined in GAP.
sage: c=H.GeneratorsOfGroup()   
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/mmarco/sage-5.4/<ipython console> in <module>()

/home/mmarco/sage-5.4/local/lib/python2.7/site-packages/sage/libs/gap/element.so in sage.libs.gap.element.GapElement.__getattr__ (sage/libs/gap/element.c:3555)()

AttributeError: Name "GeneratorsOfGroup" is not defined in GAP.
sage: H
Exception RuntimeError: 'Entered a critical block twice' in 'sage.libs.gap.util.error_handler' ignored
<fp group on the generators [ f1, f2 ]>
sage: H.GeneratorsOfGroup()
[ f1, f2 ]
sage: c=H.GeneratorsOfGroup()[0]
sage: c
f1
sage: c=H.Generat
H.GeneratorsOfField  H.GeneratorsOfGroup  H.GeneratorsOfIdeal  
sage: c=H.GeneratorsOfGroup()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/mmarco/sage-5.4/<ipython console> in <module>()

/home/mmarco/sage-5.4/local/lib/python2.7/site-packages/sage/libs/gap/element.so in sage.libs.gap.element.GapElement.__getattr__ (sage/libs/gap/element.c:3555)()

AttributeError: Name "GeneratorsOfGroup" is not defined in GAP.
sage: c=H.GeneratorsOfGroup()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)

/home/mmarco/sage-5.4/<ipython console> in <module>()

/home/mmarco/sage-5.4/local/lib/python2.7/site-packages/sage/libs/gap/element.so in sage.libs.gap.element.GapElement.__getattr__ (sage/libs/gap/element.c:3555)()

AttributeError: Name "GeneratorsOfGroup" is not defined in GAP.
sage: H
Exception RuntimeError: 'Entered a critical block twice' in 'sage.libs.gap.util.error_handler' ignored
<fp group on the generators [ f1, f2 ]>
sage: c=H.GeneratorsOfGroup()

comment:4 Changed 9 years ago by mmarco

This same behaviour appears when i don't use this patch, so reported in the libgap ticket.

comment:5 Changed 9 years ago by vbraun

  • Description modified (diff)

comment:6 Changed 9 years ago by vbraun

Updated patch fixes the issue with the command line completion

comment:7 Changed 9 years ago by vbraun

  • Cc tfeulner added

Updated spkg fixes long input lines (>32k characters):

sage: S = SymmetricGroup(10000)
sage: s = S.random_element()
sage: len(str(s._gap_()))
61673
sage: libgap(s)
(1,5243,8582,5012,6862,7477,1681,7423,8405,3101,...,5257,9362,2495,8500)( [...] )

comment:8 Changed 9 years ago by vbraun

  • Description modified (diff)

comment:9 Changed 9 years ago by vbraun

I've added the same workaround for the gap command line option issue as just discussed in #13211

comment:10 Changed 9 years ago by vbraun

I've added __nonzero__ for wrapped GAP booleans for easier use.

Changed 9 years ago by vbraun

Updated patch

comment:11 Changed 9 years ago by vbraun

The new patch allows to change the comparison method for libgap objects since not all GAP objects can be compared. This is important if you want use them as cache keys.

Also, the comparison was not properly wrapped in sig_on/sig_off

comment:12 Changed 9 years ago by vbraun

I forgot to package aclocal.m4, updated spkg adds that file.

comment:13 Changed 9 years ago by vbraun

  • Description modified (diff)

comment:14 Changed 9 years ago by vbraun

  • Description modified (diff)

comment:15 Changed 9 years ago by vbraun

Updated to libGap-4.5.7

comment:16 Changed 9 years ago by vbraun

  • Dependencies changed from #6391, #13211 to #13211
  • Description modified (diff)

Changed 9 years ago by vbraun

patch from #6391

Changed 9 years ago by vbraun

patch from #6391

comment:17 Changed 9 years ago by vbraun

  • Description modified (diff)

comment:18 Changed 9 years ago by vbraun

  • Description modified (diff)

comment:19 Changed 9 years ago by jdemeyer

  • Milestone changed from sage-5.6 to sage-5.7
  • Summary changed from Improve libGAP startup error handling to libGAP! -- create a Cython library interface to gap

comment:20 Changed 9 years ago by iandrus

  • Cc iandrus added

comment:21 Changed 9 years ago by mmarco

I still see the segfault when trying to execute "Exec" trough libgap. This seems to make it impossible to load gap packages that make use of external programs.

comment:22 Changed 9 years ago by vbraun

Thanks for reminding me, I completely forgot about the Exec thing.

In the meantime, can we get this reviewed and included into Sage before the next GAP version comes out?

comment:23 Changed 9 years ago by vbraun

  • Description modified (diff)

I've replaced the spkg with a new version that fixes the GAP Exec command.

Changed 9 years ago by vbraun

Updated patch

comment:24 Changed 9 years ago by mmarco

Thanks. Still i can't load the kbmag package :(

comment:25 Changed 9 years ago by vbraun

I've updated the spkg, now loading kbmag works for me. I had to install it by hand since your spkg at #13673 is offline.

comment:26 Changed 9 years ago by dimpase

I get a slew of errors in ptestlong after following the instructions of the ticket with Sage 5.5; e.g.:

sage -t  --long -force_lib devel/sage/sage/libs/gap/libgap.pyx
**********************************************************************
File "/usr/local/src/sage/sage-5.5/devel/sage-main/sage/libs/gap/libgap.pyx", line 11:
    sage: a = libgap(10)
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/src/sage/sage-5.5/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/local/src/sage/sage-5.5/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/local/src/sage/sage-5.5/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[2]>", line 1, in <module>
        a = libgap(Integer(10))###line 11:
    sage: a = libgap(10)
      File "lazy_import.pyx", line 244, in sage.misc.lazy_import.LazyImport.__call__ (sage/misc/lazy_import.c:2012)
      File "lazy_import.pyx", line 148, in sage.misc.lazy_import.LazyImport._get_object (sage/misc/lazy_import.c:1291)
      File "libgap.pyx", line 574, in init sage.libs.gap.libgap (sage/libs/gap/libgap.c:4157)
        libgap = Gap()
      File "libgap.pyx", line 395, in sage.libs.gap.libgap.Gap.__init__ (sage/libs/gap/libgap.c:2716)
        initialize()
      File "util.pyx", line 193, in sage.libs.gap.util.initialize (sage/libs/gap/util.c:3312)
        from sage.interfaces.gap import get_gap_memory_pool_size
    ImportError: cannot import name get_gap_memory_pool_size
**********************************************************************
File "/usr/local/src/sage/sage-5.5/devel/sage-main/sage/libs/gap/libgap.pyx", line 12:
    sage: a
Exception raised:
    Traceback (most recent call last):
      File "/usr/local/src/sage/sage-5.5/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/usr/local/src/sage/sage-5.5/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/usr/local/src/sage/sage-5.5/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_0[3]>", line 1, in <module>
        a###line 12:
    sage: a
    NameError: name 'a' is not defined

are some patches missing or not up date?

comment:27 follow-up: Changed 9 years ago by vbraun

Did you apply the patches from #13211? The get_gap_memory_pool_size() function was added in trac_13211_fix_gap_doctests_vb.patch.

Last edited 9 years ago by vbraun (previous) (diff)

comment:28 in reply to: ↑ 27 Changed 9 years ago by dimpase

Replying to vbraun:

Did you apply the patches from #13211? The get_gap_memory_pool_size() function was added in trac_13211_fix_gap_doctests_vb.patch.

indeed, I overlooked that #13211 isn't merged in 5.5. Sorry for noise.

comment:29 Changed 9 years ago by dimpase

Does sig_on() need a matching sig_off() in case sig_on() is issued in a try block, which throws an exception?

Comparison functions returning -1, 0, +1... So FORTRAN |V... :-)

comment:30 Changed 9 years ago by vbraun

sig_on and sig_off must be paired, though ideally they enclose only C/Cython code which cannot raise Python exceptions. A C signal will jump back to sig_on and raise there.

http://www.sagemath.org/doc/developer/coding_in_cython.html#using-sig-on-and-sig-off

Last edited 9 years ago by vbraun (previous) (diff)

comment:31 Changed 9 years ago by dimpase

  • Dependencies changed from #13211 to #13211, #13880

patches from #13880 are needed for doctests to (eventually) pass here.

comment:32 Changed 9 years ago by dimpase

  • Status changed from needs_review to positive_review

comment:33 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:34 Changed 9 years ago by jdemeyer

  • Description modified (diff)

comment:35 Changed 9 years ago by jdemeyer

Is the test program sage/libs/gap/test/main.c used anywhere in the Sage library? I don't mind having such a program, but I'd say it belongs in the libgap spkg and not in the Sage library. It could be run when installing libGAP.

comment:36 Changed 9 years ago by jdemeyer

  • Reviewers set to Dmitrii Pasechnik

comment:37 Changed 9 years ago by vbraun

sage/libs/gap/test/main.c is not used anywhere. As the readme file explains, it is there if you need an example for how to link with libGAP without Sage. This can be useful for debugging. LibGAP has its own testsuite.

comment:38 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

libgap tries to run various autotools, which should be avoided (as it can lead to build errors). Probably this is simply timestamps were clobbered during the packaging.

libgap-4.5.7.log:WARNING: 'aclocal-1.12' is missing on your system.  You should only need it if
libgap-4.5.7.log:WARNING: 'automake-1.12' is missing on your system.  You should only need it if
libgap-4.5.7.log:WARNING: 'autoconf' is missing on your system.  You should only need it if
libgap-4.5.7.log:WARNING: 'autoheader' is missing on your system.  You should only need it if

comment:39 Changed 9 years ago by vbraun

  • Status changed from needs_work to positive_review

Ok I've regenerated the files and checked that it builds without the WARNING on an older machine. Spkg at the same place.

comment:40 follow-up: Changed 9 years ago by jdemeyer

While building the documentation with this patch included, twice something went wrong.

  1. First time, I got an "unhandled SIGSEGV" in Sage while reading the sources of some elliptic curve file (sorry, forgot which one) for the PDF documentation. I didn't store the exact logs since I thought it was a cosmic ray or something...
  2. Next attempt, building from scratch again during the build of the HTML documentation:
    reading sources... [ 86%] sage/schemes/elliptic_curves/cm
    reading sources... [ 87%] sage/schemes/elliptic_curves/constructor
    reading sources... [ 87%] sage/schemes/elliptic_curves/ec_database
    reading sources... [ 87%] sage/schemes/elliptic_curves/ell_curve_isogeny
    reading sources... [ 87%] sage/schemes/elliptic_curves/ell_field
    reading sources... [ 87%] sage/schemes/elliptic_curves/ell_finite_field
    reading sources... [ 87%] sage/schemes/elliptic_curves/ell_generic
    
    Exception occurred:
      File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/environment.py", line 831, in read_doc
        pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
    MemoryError
    The full traceback has been saved in /tmp/release/sphinx-err-QDcECu.log, if you want to report the issue to the developers.
    

Full traceback:

# Sphinx version: 1.1.2
# Python version: 2.7.3
# Docutils version: 0.7 release
# Jinja2 version: 2.5.5
Traceback (most recent call last):
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/cmdline.py", line 189, in main
    app.build(force_all, filenames)
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/application.py", line 204, in build
    self.builder.build_update()
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/builders/__init__.py", line 196, in build_update
    'out of date' % len(to_build))
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/builders/__init__.py", line 216, in build
    purple, length):
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/builders/__init__.py", line 120, in status_iterator
    for item in iterable:
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/environment.py", line 616, in update_generator
    self.read_doc(docname, app=app)
  File "/release/merger/sage-5.7.alpha0/local/lib/python2.7/site-packages/Sphinx-1.1.2-py2.7.egg/sphinx/environment.py", line 831, in read_doc
    pickle.dump(doctree, f, pickle.HIGHEST_PROTOCOL)
MemoryError

I cannot say for sure that this ticket is to blame, but the MemoryError certainly made me suspect GAP. And I also saw some "halving pool size" messages appear during the docbuilding.

Last edited 9 years ago by jdemeyer (previous) (diff)

comment:41 in reply to: ↑ 40 ; follow-up: Changed 9 years ago by dimpase

Replying to jdemeyer:

While building the documentation with this patch included, twice something went wrong.

  1. First time, I got an "unhandled SIGSEGV" in Sage while reading the sources of some elliptic curve file (sorry, forgot which one) for the PDF documentation. I didn't store the exact logs since I thought it was a cosmic ray or something...
  2. Next attempt, building from scratch again during the build of the HTML documentation:

if this happens on one host only, I'd blame the RAM there.

comment:42 in reply to: ↑ 41 Changed 9 years ago by jdemeyer

Replying to dimpase:

if this happens on one host only, I'd blame the RAM there.

If this happens only with the libGAP patches, I'd blame libGAP. Besides, why would faulty RAM lead to a MemoryError exception?

comment:43 follow-up: Changed 9 years ago by vbraun

All GAP/libGAP processes should be terminated by the time we are building the manual, so I don't see how we could possibly impact that.

Is this a 32-bit machine? There you might run into warts caused by the limited address space, and the halving pool size would support that. Sphinx uses quite a lot of memory and might fail to load its pickles if there is no memory left or no contiguous address space block available.

comment:44 in reply to: ↑ 43 ; follow-up: Changed 9 years ago by jdemeyer

Replying to vbraun:

All GAP/libGAP processes should be terminated by the time we are building the manual

Not really, as building the manual runs Sage code (as witnessed for example that Segmentation Faults run the Sage signal handler and display "Unhandled SIGSEGV...")

comment:45 in reply to: ↑ 44 ; follow-up: Changed 9 years ago by vbraun

Replying to jdemeyer:

Not really, as building the manual runs Sage code (as witnessed for example that Segmentation Faults run the Sage signal handler and display "Unhandled SIGSEGV...")

Theoretically true, but a normal Sage session won't start GAP until it has to and libGAP is lazily imported as well. Even then, none of that has anything to do with the elliptic curve stuff.

Right now the only thing in this ticket that seems to actually influence the docbuild is that the documentation becomes larger.

comment:46 in reply to: ↑ 45 ; follow-up: Changed 9 years ago by jdemeyer

Replying to vbraun:

Theoretically true, but a normal Sage session won't start GAP until it has to and libGAP is lazily imported as well.

The fact that I'm seeing "halving pool size" messages during the docbuilding proves that GAP is started.

comment:47 in reply to: ↑ 46 Changed 9 years ago by vbraun

Replying to jdemeyer:

The fact that I'm seeing "halving pool size" messages during the docbuilding proves that GAP is started.

That indeed means that something is wonky. We are trying hard to pick a pool size that should fit into memory yet it doesn't. As I said before, this means that either there is very little memory to start with or the available address space is so fragmented that there is no contiguous piece left (32-bit only).

comment:48 Changed 9 years ago by vbraun

I guess sphinx has to resolve the lazy imports in order to get at the docstrings? That would pull in libGAP, which then subtracts ~250MB from the available address space. On a 32-bit machine that might push you over the cliff since sphinx is the most memory-demanding piece of the whole sage build process. Can you tell us which architecture we are talking about?

comment:49 Changed 9 years ago by jdemeyer

Sorry, I wasn't immediately aware of this, but this was with ulimit -v 2500000 (2.5GB) set on a 64-bit system.

comment:50 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

This needs a patch to MANIFEST.in to add the Makefile (but again: I doubt that this really belongs in the Sage library).

Changed 9 years ago by vbraun

Initial patch

comment:51 Changed 9 years ago by vbraun

  • Description modified (diff)
  • Status changed from needs_work to positive_review

I've added it to the manifest. I do think it is useful, when I started working on libgap that was one of the first things that I had to figure out.

Changed 9 years ago by jdemeyer

comment:52 Changed 9 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from positive_review to needs_work

comment:53 Changed 9 years ago by jdemeyer

  • Status changed from needs_work to needs_review

New SAGE_ROOT patch which adds libgap as dependency of the Sage library.

comment:54 Changed 9 years ago by vbraun

  • Status changed from needs_review to positive_review

Looks good to me!

Changed 9 years ago by jdemeyer

Patch needed for SPARC

comment:55 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

libgap_sparc.patch must be added to the spkg to work on SPARC.

comment:56 Changed 9 years ago by vbraun

Thanks. I've added it to the symbol rewriting code in the make_spkg.sh script.

#ifdef SPARC
#if SPARC
        asm("           .globl  SparcStackFuncBags              ");
        asm("   SparcStackFuncBags:                             ");
        asm("           ta      0x3     ! ST_FLUSH_WINDOWS      ");
        asm("           mov     %sp,%o0                         ");
        asm("           retl                                    ");
        asm("           nop                                     ");
#endif
#endif

/* sunos */
#ifdef SPARC
#if SPARC
        asm("           .globl  _SparcStackFuncBags             ");
        asm("   _SparcStackFuncBags:                            ");
        asm("           ta      0x3     ! ST_FLUSH_WINDOWS      ");
        asm("           mov     %sp,%o0                         ");
        asm("           retl                                    ");
        asm("           nop                                     ");
#endif
#endif
#else
#if defined(SPARC) && SPARC
void SparcStackFuncBags( void )
{
  asm (" ta 0x3 ");
  asm (" mov %sp,%o0" );
  return;
}
#endif

I presume the offending code has grown organically over the decades...

comment:57 Changed 9 years ago by vbraun

  • Status changed from needs_work to positive_review

comment:58 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

The similar operation needs to be done for

ItaniumRegisterStackTop

comment:59 Changed 9 years ago by jdemeyer

Also: please remove src/src/gasman.c.orig

comment:60 follow-up: Changed 9 years ago by vbraun

Yay, more conditional compilation ;-) I've added the conditional itanium.s source. I also changed the make_spkg.sh script to start from the tarball, so no superfluous files should be included. I tested the new spkg (same location) successfully on sage.math and iras.

comment:61 Changed 9 years ago by vbraun

  • Status changed from needs_work to positive_review

comment:62 in reply to: ↑ 60 Changed 9 years ago by jdemeyer

Replying to vbraun:

I tested the new spkg

I assume that also includes running libgap doctests, because that's where the error shows up.

comment:63 Changed 9 years ago by vbraun

Well I checked that itanium.s is compiled and linked into libgap on iras.

comment:64 Changed 9 years ago by vbraun

I compiled sage on iras overnight and doctests pass, too ;-)

comment:65 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

Something went wrong with your search/replace script, on mark (Solaris):

ld.so.1: python2.7: fatal: relocation error: file /home/buildbot/build/sage/mark-1/mark_full/build/sage-5.7.beta0/local/lib/libgap.so.0: symbol libGAP_libGAP_SparcStackFuncBags: referenced symbol not found

comment:66 Changed 9 years ago by SimonKing

  • Cc SimonKing added

comment:67 Changed 9 years ago by vbraun

  • Status changed from needs_work to positive_review

Fixed - the SparcStackFuncBags occurs in inline assembly and in C, so I now made sure to only rename it once.

comment:68 Changed 9 years ago by jdemeyer

  • Status changed from positive_review to needs_work

I'm still not entirely happy with the memory allocation during docbuilding. For reasons I cannot understand, the documentation used to build fine with

ulimit -v 2500000

but now it needs (I tested 100MB increments)

ulimit -v 3500000

which is an increase of 1GB. Of course, the documentation is also getting bigger, but that explains only a small part of the increase. Also, on the buildbot I'm seeing sometimes errors of the form

sphinx-build -b html -d /home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/devel/sage/doc/output/doctrees/de/tutorial    /home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/devel/sage/doc/de/tutorial /home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/devel/sage/doc/output/html/de/tutorial
Traceback (most recent call last):
  File "/home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/devel/sage/doc/common/builder.py", line 1093, in <module>
    getattr(get_builder(name), type)()
  File "/home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/devel/sage/doc/common/builder.py", line 232, in _wrapper
    getattr(get_builder(document), name)(*args, **kwds)
  File "/home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/devel/sage/doc/common/builder.py", line 88, in f
    subprocess.call(build_command, shell=True)
  File "/home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/local/lib/python/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/local/lib/python/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/home/buildbot/build/sage/iras-1/iras_full/build/sage-5.7.beta0/local/lib/python/subprocess.py", line 1143, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
make: *** [doc-html-mathjax] Error 1

I suspect that GAP has something to do with this.

Could you somehow fix the memory allocation to allocate an absolutely minimal amount during docbuilding? Maybe it would make sense to introduce an environment variable SAGE_GAP_MEGABYTES which could be user-set to the desired memory size and which would be set by the docbuilder to a small number (say, 1 MB).

Last edited 9 years ago by jdemeyer (previous) (diff)

comment:69 Changed 9 years ago by vbraun

  • Description modified (diff)
  • Status changed from needs_work to positive_review

I've added a setting to our Sphinx config (doc/common/conf.py) and builder config (build_options.py) to use only the minimum amount for GAP memory pool. The builder and sphinx itself both import everything to get at docstrings, so two libgap instances are being created when building the documentation. I verified that I can now build all documentation with ulimit -v 2500000).

comment:70 Changed 9 years ago by jdemeyer

The following is fairly reproducible:

sage -t  -force_lib devel/sagenb-main/sagenb/misc/support.py
**********************************************************************
File "/release/merger/sage-5.7.beta1/devel/sagenb-main/sagenb/misc/support.py", line 479:
    sage: sage.server.support.syseval(gap, "2+3")
Expected:
    '5'
Got:
    ** Gap crashed or quit executing '2+3;' **
    Restarting Gap and trying again
    '5'
**********************************************************************

Note that GAP is started with only 1MB of memory, as the top of this strace log shows:

execve("/release/merger/sage-5.7.beta1/local/gap/latest/bin/x86_64-unknown-linux-gnu-gcc-default64/gap", ["/release/merger/sage-5.7.beta1/local/gap/latest/bin/x86_64-unknown-linux-gnu-gcc-default64/gap", "-m", "24m", "-l", "/release/merger/sage-5.7.beta1/local/gap/latest", "-r", "-b", "-p", "-T", "-o", "1m", "-s", "1m", "/release/merger/sage-5.7.beta1/local/share/sage/ext/gap/sage.g"], [/* 84 vars */]) = 0

Full strace log: http://boxen.math.washington.edu/home/jdemeyer/gap.28216

Version 1, edited 9 years ago by jdemeyer (previous) (next) (diff)

Changed 9 years ago by vbraun

Updated patch

comment:71 Changed 9 years ago by vbraun

I see, introspection runs Sphinx as well so there is a chance to get conf.py imported in Sage. Though curiously, it doesn't cause any doctest failure on my machine. I moved the part that limits GAP to a branch in conf.py that is only executed when we actually build the documentation, this should fix it.

Last edited 9 years ago by vbraun (previous) (diff)

comment:72 Changed 9 years ago by jdemeyer

  • Merged in set to sage-5.7.beta1
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:73 Changed 9 years ago by jdemeyer

  • Description modified (diff)
Note: See TracTickets for help on using tickets.