Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#8192 closed enhancement (fixed)

Update PolyBoRi to newest upstream release

Reported by: malb Owned by: PolyBoRi
Priority: major Milestone: sage-4.4.1
Component: packages: standard Keywords:
Cc: PolyBoRi, burcin Merged in: sage-4.4.1.alpha1
Authors: Alexander Dreyer, Burcin Erocal Reviewers: Martin Albrecht
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by malb)

  • gat parser added
  • Disabled keyboard interrupt handling (but re-raised)
  • Fixed critical bug in normal form
  • Naming convention: minimalElements -> minimal_elements
  • has_constant_part for variable/monomial
  • lead/lex_lead/lead_deg/lex_lead_deg also for Variable/Monomial?
  • iterator for literal factorization
  • Added treatment of customizable settings for BOOST_LIBRARY,SHCFLAGS, SHCCFLAGS, and SHCXXFLAGS
  • Improved Sun Studio compatibility
  • Fix for hpux (CUDD needs pwd.h)

This should be relatively straight forward then.

Change History (34)

comment:1 Changed 11 years ago by malb

  • Description modified (diff)

comment:2 Changed 11 years ago by PolyBoRi

  • Cc burcin added
  • Status changed from new to needs_review

Burcin and I prepared the Package:

http://sage.math.washington.edu/home/dreyer/spkg/polybori-0.6.4.spkg

Regards,

Alexander Dreyer

comment:3 Changed 11 years ago by PolyBoRi

  • Owner changed from tbd to PolyBoRi

comment:4 Changed 11 years ago by PolyBoRi

  • Authors set to PolyBoRi,burcin

comment:5 Changed 11 years ago by burcin

  • Authors changed from PolyBoRi,burcin to Alexander Dreyer, Burcin Erocal

comment:6 Changed 11 years ago by malb

  • Status changed from needs_review to positive_review

Looks good.

comment:7 Changed 11 years ago by jhpalmieri

Do you know why the spkg is so small?

palmieri@sage:~$ ls -l /home/dreyer/spkg/polybori-0.6.4.spkg 
... 2040576 2010-03-12 06:13 /home/dreyer/spkg/polybori-0.6.4.spkg
palmieri@sage:~$ ls -l /home/release/sage-4.3.5/sage-4.3.5/spkg/standard/polybori-0.6.3-20091028.spkg 
... 6825939 2010-02-11 08:56 /home/release/sage-4.3.5/sage-4.3.5/spkg/standard/polybori-0.6.3-20091028.spkg

I'm not complaining, but I'm curious.

comment:8 Changed 11 years ago by PolyBoRi

Indeed, the polybori-0-6-3*spkg's contain a clone of PolyBoRi?'s mercurial repository as well a a mercurial repository of the Sage-related patches, The polybori-0-6-4.spkg only has the patch repo.

What is Sage's policy here?

Best regards,

Alexander

comment:9 Changed 11 years ago by jhpalmieri

I don't think there is a policy, but it seems to be okay to delete an upstream mercurial repository like this.

comment:10 Changed 11 years ago by mhansen

  • Reviewers set to Martin Albrecht

This does not build on Cygwin due to missing libpng12 and libz. I've posted an spkg which fixes this at

http://sage.math.washington.edu/home/mhansen/cygwin_port/polybori-0.6.4.spkg

Could someone review my changes? Otherwise, I can make a new ticket for this.

comment:11 Changed 11 years ago by malb

Hi Mike,

I'm afraid your SPKG will need a new ticket since this one was merged in 4.4 already.

Your SPKG builds fine on sage.math, however I get

Exiting Sage (CPU time 0m0.06s, Wall time 0m0.88s).
*** glibc detected *** python: corrupted double-linked list: 0x000000000329b9c0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7fccf4577663]
/lib/libc.so.6[0x7fccf4578ea1]
/lib/libc.so.6(cfree+0x8c)[0x7fccf457cc1c]
/lib/libc.so.6(exit+0xe0)[0x7fccf453a110]
python[0x4baac3]
python(PyErr_PrintEx+0x19a)[0x4bacca]
python(PyRun_SimpleFileExFlags+0x116)[0x4bb926]
python(Py_Main+0x984)[0x416354]
/lib/libc.so.6(__libc_start_main+0xf4)[0x7fccf45231c4]
python[0x415629]

which is likely unrelated to your SPKG? I will build a fresh 4.4 to check.

comment:12 Changed 11 years ago by mhansen

Okay, I'll open a new ticket. Shouldn't this ticket have been closed when it was merged into 4.4?

comment:13 Changed 11 years ago by malb

Argh, I mixed it up. No it isn't merged, so no new ticket needed.

comment:14 Changed 11 years ago by malb

SPKG looks good, the bug reported above had nothing to do with the SPKG. Running doctests now, if they pass this should be a positive review.

comment:15 Changed 11 years ago by mhansen

Thanks!

comment:16 Changed 11 years ago by malb

Mhh, can you try your SPKG on sage.math? It seems it does produce a fair amount of segfaults.

comment:17 Changed 11 years ago by mhansen

Sure.

comment:18 follow-up: Changed 11 years ago by was

  • Status changed from positive_review to needs_work

given the report of segfaults above, I'm changing this to needs work!

comment:19 in reply to: ↑ 18 Changed 11 years ago by burcin

Replying to was:

given the report of segfaults above, I'm changing this to needs work!

Or you could merge the package which got a positive review and we move the Cygwin problems to a different ticket.

http://sage.math.washington.edu/home/dreyer/spkg/polybori-0.6.4.spkg

comment:20 Changed 11 years ago by mhansen

  • Status changed from needs_work to needs_review

Yep, I think using that one would be fine for now. I'll look into these segfaults and make a new ticket.

comment:21 Changed 11 years ago by mhansen

  • Status changed from needs_review to positive_review

comment:22 Changed 11 years ago by was

  • Merged in set to 4.4.1.alpha1
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:23 Changed 11 years ago by wjp

Both the 0.6.4.spkgs here give those crashes reported above, but only once in every 10-15 runs.

Valgrind shows invalid reads/writes on exit pointing to polybori:

==14880== Invalid read of size 4
==14880==    at 0x30CE3642: __tcf_1 (sp_counted_base_gcc_x86.hpp:50)
==14880==    by 0x56FEB48: exit (in /lib64/libc-2.9.so)
==14880==    by 0x4B9531: handle_system_exit (pythonrun.c:1716)
==14880==    by 0x4B9738: PyErr_PrintEx (pythonrun.c:1126)
==14880==    by 0x4BA334: PyRun_SimpleFileExFlags (pythonrun.c:1035)
==14880==    by 0x4164C4: Py_Main (main.c:142)
==14880==    by 0x56E95E3: (below main) (in /lib64/libc-2.9.so)
==14880==  Address 0x3225a538 is not stack'd, malloc'd or (recently) free'd
==14880== 
==14880== Invalid read of size 8
==14880==    at 0x30CE365C: __tcf_1 (CCuddCore.h:208)
==14880==    by 0x56FEB48: exit (in /lib64/libc-2.9.so)
==14880==    by 0x4B9531: handle_system_exit (pythonrun.c:1716)
==14880==    by 0x4B9738: PyErr_PrintEx (pythonrun.c:1126)
==14880==    by 0x4BA334: PyRun_SimpleFileExFlags (pythonrun.c:1035)
==14880==    by 0x4164C4: Py_Main (main.c:142)
==14880==    by 0x56E95E3: (below main) (in /lib64/libc-2.9.so)
==14880==  Address 0x31a599e8 is 8 bytes inside a block of size 64 free'd
==14880==    at 0x4C2216E: operator delete(void*) (vg_replace_malloc.c:346)
==14880==    by 0x56FEB48: exit (in /lib64/libc-2.9.so)
==14880==    by 0x4B9531: handle_system_exit (pythonrun.c:1716)
==14880==    by 0x4B9738: PyErr_PrintEx (pythonrun.c:1126)
==14880==    by 0x4BA334: PyRun_SimpleFileExFlags (pythonrun.c:1035)
==14880==    by 0x4164C4: Py_Main (main.c:142)
==14880==    by 0x56E95E3: (below main) (in /lib64/libc-2.9.so)
==14880== 
==14880== Invalid write of size 8
==14880==    at 0x30CE3667: __tcf_1 (CCuddCore.h:208)
==14880==    by 0x56FEB48: exit (in /lib64/libc-2.9.so)
==14880==    by 0x4B9531: handle_system_exit (pythonrun.c:1716)
==14880==    by 0x4B9738: PyErr_PrintEx (pythonrun.c:1126)
==14880==    by 0x4BA334: PyRun_SimpleFileExFlags (pythonrun.c:1035)
==14880==    by 0x4164C4: Py_Main (main.c:142)
==14880==    by 0x56E95E3: (below main) (in /lib64/libc-2.9.so)
==14880==  Address 0x31a599e8 is 8 bytes inside a block of size 64 free'd
==14880==    at 0x4C2216E: operator delete(void*) (vg_replace_malloc.c:346)
==14880==    by 0x56FEB48: exit (in /lib64/libc-2.9.so)
==14880==    by 0x4B9531: handle_system_exit (pythonrun.c:1716)
==14880==    by 0x4B9738: PyErr_PrintEx (pythonrun.c:1126)
==14880==    by 0x4BA334: PyRun_SimpleFileExFlags (pythonrun.c:1035)
==14880==    by 0x4164C4: Py_Main (main.c:142)
==14880==    by 0x56E95E3: (below main) (in /lib64/libc-2.9.so)

comment:24 follow-up: Changed 11 years ago by burcin

Hi Willem Jan,

The crashes are probably caused by the fact that I disabled the code that statically links polybori in the new package. IIRC, we were doing that exactly because of these double free errors. We tested the new package on different platforms, and didn't see the crashes, so we assumed the problem was fixed.

I don't have time to reproduce the problem, or test new packages at the moment. Can you try enabling the static build and try again?

From the diff of the changes (of an earlier version of the package in Alexander Dreyer's home dir), this looks like the relevant change:

diff --git a/patches/SConstruct b/patches/SConstruct
--- a/patches/SConstruct
+++ b/patches/SConstruct
@@ -561,7 +581,7 @@
 gb_src = [GBPath('src', source) for source in gb_src]
 if not(external_m4ri):
    gb_src += m4ri
-gb=env.StaticLibrary(GBPath('groebner'), gb_src+[libpb])
+gb=env.StaticLibrary(GBPath('groebner'), gb_src)#+[libpb])
 
 #print "gb:", gb, dir(gb)
 #sometimes l seems to be boxed by a list

BTW, this also looks suspect to me:

diff --git a/patches/SConstruct b/patches/SConstruct
--- a/patches/SConstruct
+++ b/patches/SConstruct
@@ -327,7 +350,7 @@
 Help(opts.GenerateHelpText(env))
 
 have_l2h = have_t4h = False
-external_m4ri = True
+external_m4ri = False
 
 if not env.GetOption('clean'):
     conf = Configure(env)

Many thanks for looking into this.

comment:25 Changed 11 years ago by PolyBoRi

Hi Burcin,

I think, we removed the call of remove_dylib  from spkg-install. So reintroducing remove_dylib may fit it already.

Regards,

Alexander Dreyer

comment:26 in reply to: ↑ 24 Changed 11 years ago by malb

BTW, this also looks suspect to me:

diff --git a/patches/SConstruct b/patches/SConstruct
--- a/patches/SConstruct
+++ b/patches/SConstruct
@@ -327,7 +350,7 @@
 Help(opts.GenerateHelpText(env))
 
 have_l2h = have_t4h = False
-external_m4ri = True
+external_m4ri = False
 
 if not env.GetOption('clean'):
     conf = Configure(env)

I agree, that this wrong and should be reverted (we really want the external M4RI)

comment:27 Changed 11 years ago by PolyBoRi

Hi malb,

right, but this is only the inital value of an internal variable. Later in the Sconstruct file, external_m4ri will be set to True, if the library is found.

Regards,

Alexander Dreyer

comment:28 Changed 11 years ago by wjp

I don't have time to extensively test right now, so I tried both reverting the patch that Burcin gave above and reintroducing remove_dylib at the same time. That fixed the problems.

So you're definitely on the right track, but I'll leave it to you to figure out what exactly needs to be changed for a new spkg :-)

comment:29 Changed 11 years ago by PolyBoRi

Hi Willem Jan,

I changed than remove_dynlib only at:

http://sage.math.washington.edu/home/dreyer/spkg/polybori-0.6.4-p1.spkg

Is this enough to fix the problem?

Since I was not able to reproduce the problem: Can you test it? (How do you run it?)

Best regards,

Alexander Dreyer

comment:30 Changed 11 years ago by wjp

Hi Alexander,

Yes, I don't get the errors anymore with that package. (After renaming the spkg to polybori-0.6.4.p1.spkg and the directory inside it to match the spkg filename.)

I test it by running sage through valgrind and seeing if there are any errors on exit in polybori files, since the actual crashes happen here only once every 10-15 runs or so.

Still I can't help but wonder if there's something subtle going wrong with sage's interface to polybori or in polybori itself on exit to require static linking. Maybe something like multiple libraries sharing a global memory manager and each of them trying to clean up global structures on quit or something similar...

Thanks! -Willem Jan

comment:31 Changed 11 years ago by mvngu

From sage-devel:

Sage 4.4.1.alpha2 contains two PolyBoRi spkg's:

  • polybori-0.6.3-20091028.spkg
  • polybori-0.6.4.spkg

I think polybori-0.6.4.spkg is the newer one, so I deleted the other one from SAGE_ROOT/spkg/standard/. Here's a diff between the spkg-install of polybori-0.6.3-20091028.spkg and polybori-0.6.4.spkg:

  • spkg-install

    [mvngu@sage mvngu]$ diff -Naur polybori-0.6.3-20091028/spkg-install polybori-0.6.4.p0/spkg-install 
    old new  
    66   exit 1
    77fi
    88
    9 PBDIR=polybori-0.6
     9PBDIR=polybori-0.6.4
    1010WORKDIR=${PWD}/src
    1111SCONS=scons
    1212
    1313# For some strange reason the installed boost in $SAGE_LOCAL causes
    1414# make install failures, so copy it over.
    15 mkdir src/boost_1_34_1.cropped
    16 cp -r $SAGE_LOCAL/include/boost src/boost_1_34_1.cropped
     15#mkdir src/boost_1_34_1.cropped
     16#cp -r $SAGE_LOCAL/include/boost src/boost_1_34_1.cropped
    1717BOOSTDIR=boost_1_34_1.cropped
    1818
    1919patch()
     
    2626
    2727    cp patches/SConstruct src/${PBDIR}/SConstruct
    2828    cp patches/PyPolyBoRi.py src/${PBDIR}/pyroot/polybori
    29 
    30     # workaround so will build on cygwin
    31     cp patches/cpu_stats.c src/${PBDIR}/Cudd/util/cpu_stats.c
    3229}
    3330
    3431
     
    6865
    6966remove_dylib()
    7067{
    71     # linking dynmic libraries causes segfaults at exit (see #2822)
     68    # linking dynamic libraries causes segfaults at exit (see #2822)
    7269    if [ `uname` = "Darwin" ]; then
    7370        rm -f $SAGE_LOCAL/lib/libpolybori.dylib
    7471        rm -f $SAGE_LOCAL/lib/libpboriCudd.dylib
     
    10198echo "Removing dynamic libraries..."
    10299remove_dylib
    103100echo "Done removing dynamic libraries."
    104 
    105 # force a rebuild of the PolyBoRi extension
    106 if [ -f $SAGE_ROOT/devel/sage/sage/rings/polynomial/pbori.pyx ]; then
    107     touch $SAGE_ROOT/devel/sage/sage/rings/polynomial/pbori.pyx
    108 fi
    109 

I replaced polybori-0.6.4.spkg with

http://sage.math.washington.edu/home/mvngu/spkg/standard/polybori/polybori-0.6.4.p0.spkg

The latter spkg restores the command "remove_dynlib". I then built Sage 4.4.1.alpha2 from scratch on sage.math with polybori-0.6.4.p0.spkg. Doctesting resulted in the following failure:

[mvngu@sage sage-4.4.1.alpha2]$ ./sage -t -long local/lib/python2.6/site-packages/sagenb-0.8-py2.6.egg/sagenb/misc/misc.py
sage -t -long "local/lib/python2.6/site-packages/sagenb-0.8-py2.6.egg/sagenb/misc/misc.py"
**********************************************************************
File "/dev/shm/mvngu/sandbox/sage-4.4.1.alpha2/local/lib/python2.6/site-packages/sagenb-0.8-py2.6.egg/sagenb/misc/misc.py", line 109:
    sage: print "ignore this";  sage.server.misc.find_next_available_port('', 9000, verbose=False)   # random output -- depends on network
Exception raised:
    Traceback (most recent call last):
      File "/dev/shm/mvngu/sandbox/sage-4.4.1.alpha2/local/bin/ncadoctest.py", line 1231, in run_one_test
        self.run_one_example(test, example, filename, compileflags)
      File "/dev/shm/mvngu/sandbox/sage-4.4.1.alpha2/local/bin/sagedoctest.py", line 38, in run_one_example
        OrigDocTestRunner.run_one_example(self, test, example, filename, compileflags)
      File "/dev/shm/mvngu/sandbox/sage-4.4.1.alpha2/local/bin/ncadoctest.py", line 1172, in run_one_example
        compileflags, 1) in test.globs
      File "<doctest __main__.example_2[2]>", line 1, in <module>
        print "ignore this";  sage.server.misc.find_next_available_port('', Integer(9000), verbose=False)   # random output -- depends on network###line 109:
    sage: print "ignore this";  sage.server.misc.find_next_available_port('', 9000, verbose=False)   # random output -- depends on network
      File "/dev/shm/mvngu/sandbox/sage-4.4.1.alpha2/local/lib/python/site-packages/sage/server/misc.py", line 105, in find_next_available_port
        for port in range(start, start+max_tries+1):
      File "element.pyx", line 1271, in sage.structure.element.RingElement.__add__ (sage/structure/element.c:10830)
      File "coerce.pyx", line 765, in sage.structure.coerce.CoercionModel_cache_maps.bin_op (sage/structure/coerce.c:6966)
    TypeError: unsupported operand parent(s) for '+': '<type 'str'>' and 'Integer Ring'

I have wrapped up a sage.math binary of this patched version of Sage 4.4.1.alpha2. You can find it at

http://sage.math.washington.edu/home/mvngu/sage.math-bin/sage-4.4.1.alpha2-patched-sage.math.washington.edu-x86_64-Linux.tar.gz

I removed the directory

SAGE_ROOT/devel/sage-main/doc/en/thematic_tutorials/

and wrapped up a source distribution, which can be found at

http://sage.math.washington.edu/home/mvngu/sage-src/sage-4.4.1.alpha2-patched.tar

comment:32 Changed 11 years ago by wjp

I'm wondering if we should create a new ticket for this new polybori p0 spkg. In any case, I tested it, and it works fine for me.

comment:33 Changed 11 years ago by wjp

I created ticket #8830 for the p0 spkg.

comment:34 Changed 11 years ago by mvngu

  • Merged in changed from 4.4.1.alpha1 to sage-4.4.1.alpha1
Note: See TracTickets for help on using tickets.