Opened 7 years ago

Closed 7 years ago

#14323 closed defect (fixed)

libGAP messes with Python subprocesses

Reported by: jdemeyer Owned by: was
Priority: blocker Milestone: sage-5.9
Component: interfaces Keywords: child processes
Cc: vbraun, leif, roed Merged in: sage-5.9.beta2
Authors: Volker Braun Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

I haven't figured out when/why, but this might happen:

sage -t devel/sage/sage/homology/simplicial_complex.py
**********************************************************************
File "devel/sage/sage/homology/simplicial_complex.py", line 3134, in sage.homology.simplicial_complex.SimplicialComplex.automorphism_group
Failed example:
    S.automorphism_group().is_isomorphic(SymmetricGroup(4))
Exception raised:
    Traceback (most recent call last):
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 455, in _run
        self.execute(example, compiled, test.globs)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 811, in execute
        exec compiled in globs
      File "<doctest sage.homology.simplicial_complex.SimplicialComplex.automorphism_group[1]>", line 1, in <module>
        S.automorphism_group().is_isomorphic(SymmetricGroup(Integer(4)))
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/homology/simplicial_complex.py", line 3155, in automorphism_group
        [f.tuple() for f in self.facets()]])
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/graphs/generic_graph.py", line 16414, in automorphism_group
        output.append(PermutationGroup([perm_group_elt(aa) for aa in a]))
      File "refinement_graphs.pyx", line 941, in sage.groups.perm_gps.partn_ref.refinement_graphs.perm_group_elt (sage/groups/perm_gps/partn_ref/refinement_graphs.c:32796)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/groups/perm_gps/permgroup.py", line 653, in __call__
        return self._element_class()(x, self, check=check)
      File "permgroup_element.pyx", line 452, in sage.groups.perm_gps.permgroup_element.PermutationGroupElement.__init__ (sage/groups/perm_gps/permgroup_element.c:4490)
      File "sage_object.pyx", line 474, in sage.structure.sage_object.SageObject._gap_ (sage/structure/sage_object.c:4544)
      File "sage_object.pyx", line 450, in sage.structure.sage_object.SageObject._interface_ (sage/structure/sage_object.c:4144)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 199, in __call__
        return cls(self, x, name=name)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/expect.py", line 1280, in __init__
        self._name = parent._create(value, name=name)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/interface.py", line 389, in _create
        self.set(name, value)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1322, in set
        out = self._eval_line(cmd, allow_use_file=True)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 730, in _eval_line
        self._start()
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1170, in _start
        gap_reset_workspace(verbose=False)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1496, in gap_reset_workspace
        g = Gap(use_workspace_cache=False, max_workspace_size=None)
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 1084, in __init__
        max_workspace_size = _get_gap_memory_pool_size_MB()
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 292, in _get_gap_memory_pool_size_MB
        pool = get_gap_memory_pool_size()
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/interfaces/gap.py", line 268, in get_gap_memory_pool_size
        suggested_size = min(suggested_size, int(mem.virtual_memory_limit()/10))
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python2.7/site-packages/sage/misc/memory_info.py", line 131, in virtual_memory_limit
        if platform.architecture()[0] == '32bit':
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python/platform.py", line 1084, in architecture
        output = _syscmd_file(executable, '')
      File "/padic/release/merger/sage-5.9.beta0/local/lib/python/platform.py", line 1032, in _syscmd_file
        rc = f.close()
    IOError: [Errno 10] No child processes
**********************************************************************

Looking at a strace, this is because libGAP installs a SIGCHLD signal handler which interferes with Python's multiprocessing.

spkg: http://boxen.math.washington.edu/home/jdemeyer/spkg/libgap-4.5.7.p2.spkg

(signal handler fix only, see https://bitbucket.org/vbraun/libgap/commits/3f4ebd64fc1219858802173cd86b7910040fef36)

Change History (19)

comment:1 Changed 7 years ago by jdemeyer

  • Summary changed from libGAP messes with Python multiprocessing to libGAP messes with Python subprocesses

comment:2 Changed 7 years ago by jdemeyer

Does Sage support libGAP child processes? Otherwise, that feature could simply be removed from libGAP.

comment:3 Changed 7 years ago by leif

  • Cc leif added
  • Keywords child processes added

comment:4 follow-up: Changed 7 years ago by vbraun

  • Authors set to Volker Braun
  • Milestone changed from sage-5.9 to sage-duplicate/invalid/wontfix
  • Status changed from new to needs_review

I removed the SIGCHLD handler already, you just need to review #14039.

Close this ticket as duplicate.

comment:5 in reply to: ↑ 4 ; follow-up: Changed 7 years ago by jdemeyer

  • Milestone changed from sage-duplicate/invalid/wontfix to sage-5.9
  • Status changed from needs_review to needs_work

Replying to vbraun:

Close this ticket as duplicate.

Not really. I'd like to fix this as soon as possible, which means without waiting for #14187+#14014+#14039.

comment:6 in reply to: ↑ 5 Changed 7 years ago by vbraun

Replying to jdemeyer:

Not really. I'd like to fix this as soon as possible, which means without waiting for #14187+#14014+#14039.

Well tough, I would like those tickets reviewed as soon as possible because its an immense maintenance burden for me to carry around a patch that rebases all matrix groups.

comment:7 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_work to positive_review

comment:8 Changed 7 years ago by jdemeyer

  • Status changed from positive_review to needs_work
  • Work issues set to SPKG.txt

Volker: you are missing an entry in SPKG.txt for this version. Should I add it manually or will you do it using your scripts?

comment:9 follow-up: Changed 7 years ago by vbraun

comment:10 in reply to: ↑ 9 Changed 7 years ago by jdemeyer

Replying to vbraun:

I've added the SPKG entry in http://www.stp.dias.ie/~vbraun/Sage/spkg/libgap-4.6.2.p0.spkg

...which is the wrong version. This ticket is about libgap-4.5.7.p2.

comment:11 Changed 7 years ago by vbraun

Then I suggest we close this one as superseded by #14039.

comment:12 Changed 7 years ago by jdemeyer

Volker, I don't understand what you're trying to accomplish here. Acting like you're doing now is not helping. As I already said, this is an important issue which cannot wait for #14039.

I can totally understand your frustration that nobody is reviewing #14014, but this ticket shouldn't suffer from that. If you want reviewers for #14014, you could either ask David Roe (which is listed as reviewer) or ask sage-devel.

comment:13 follow-up: Changed 7 years ago by vbraun

I don't understand why it is an issue to fix this blocker before sage-5.9 is released. We are currently at beta0, so it seems there will be at least 1-2 months until this needs to be fixed.

comment:14 in reply to: ↑ 13 Changed 7 years ago by vbraun

Sorry, wanted to say: I don't understand why it is an issue to fix this blocker long before sage-5.9 is released. We are currently at beta0, so it seems there will be at least 1-2 months until this needs to be fixed. If #14039 isn't reviewed by rc0 we can talk.

Last edited 7 years ago by vbraun (previous) (diff)

comment:15 follow-up: Changed 7 years ago by jdemeyer

Why wait for rc0 to fix this? We know the problem and we know the fix.

Imagine we don't fix this before sage-5.9.rc0. The problem on this ticket is causing random strange failures, so for every multiprocessing-related problem that people report, we would need to check whether this ticket fixes it.

comment:16 in reply to: ↑ 15 Changed 7 years ago by vbraun

  • Cc roed added

Replying to jdemeyer:

Why wait for rc0 to fix this? We know the problem and we know the fix.

And the fix is at #14039.

Imagine we don't fix this before sage-5.9.rc0. The problem on this ticket is causing random strange failures, so for every multiprocessing-related problem that people report, we would need to check whether this ticket fixes it.

I'll be happy to tell every one of them that this is their own fault for not reviewing #14039.

Maybe David can tell us how long it'll take...

comment:17 Changed 7 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to positive_review
  • Work issues SPKG.txt deleted

Fixed SPKG.txt.

comment:18 Changed 7 years ago by leif

FWIW, I've built and tested Sage 5.9.beta0 from scratch with this spkg without issues, but I didn't have problems before either. (Ubuntu 10.04.4 LTS x86_64)

comment:19 Changed 7 years ago by jdemeyer

  • Merged in set to sage-5.9.beta2
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.