Opened 7 years ago

# LibGAP exits when out of memory and takes Sage with it

Reported by: Owned by: vbraun joyner major sage-8.2 group theory gap N/A #22626

The original example is no longer valid as of Sage 8.2-beta6. The following works nicely and the reason for why it was not working for Sage 5 and 6 was apparently never found.

This is correct and expected behaviour:

```sage: G=FreeGroup(2)
sage: G.inject_variables()
sage: H=G.quotient([x0*x1^2])
sage: H.as_permutation_group()
Defining x0, x1
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-2fcc9750489e> in <module>()
2 G.inject_variables()
3 H=G.quotient([x0*x1**Integer(2)])
----> 4 H.as_permutation_group()

/home/vit/bin/sage-dev/local/lib/python2.7/site-packages/sage/groups/finitely_presented.pyc in as_permutation_group(self, limit)
1008                 coset_table = self.gap().CosetTable(trivial_subgroup).sage()
1009             except ValueError:
-> 1010                 raise ValueError('Coset enumeration exceeded limit, is the group finite?')
1011         from sage.combinat.permutation import Permutation
1012         from sage.groups.perm_gps.permgroup import PermutationGroup

ValueError: Coset enumeration exceeded limit, is the group finite?
```

But the problem persists in some form as the following (real life) example shows.

```sage: k = 3
sage: p = 18
sage: d = 2*p/(p-6)
sage: l = 2*p*k/(p*k-2*k-2*p)
sage: F.<J,P,R1,R2> = FreeGroup()
sage: G = F / [J^3, P^(3*d), R1^p, R2^p, (F([-2,1]))^k, (F([4,3,1]))^l, F([-4,2,3,-2]), F([-4,1,3,-1]), F([-2,3,4])]
sage: k = G.rewriting_system()
sage: x = k.reduce(P^8)
sage: y = k.reduce(G([-4,2,3]))
sage: t = k.reduce(x*y)
sage: t == G.one()
gap: cannot extend the workspace any more!
vit@HP-EliteBook-8460p:~/bin/sage-dev\$
```

Sage should gracefully recover from errors and crashes of external libraries. (Moreover, the notebook interface doesn't give you any clue as to what went wrong. In this case you don't even know that it was GAP who caused the crash.)

### comment:1 Changed 7 years ago by vbraun

Actually this is in Sage-5.11.beta1 which still has libgap-4.5.7. Something in Sage changed, not libgap.

### comment:2 Changed 7 years ago by jdemeyer

• Milestone changed from sage-5.11 to sage-5.12

### comment:3 Changed 6 years ago by vbraun_spam

• Milestone changed from sage-6.1 to sage-6.2

### comment:4 Changed 6 years ago by vbraun_spam

• Milestone changed from sage-6.2 to sage-6.3

### comment:5 Changed 6 years ago by vbraun_spam

• Milestone changed from sage-6.3 to sage-6.4

### comment:6 follow-up: ↓ 9 Changed 2 years ago by vittucek

For Sage 8.2-beta6 I get nice error exception.

```---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-599244da5541> in <module>()
2 G.inject_variables()
3 H=G.quotient([x0*x1**Integer(2)])
----> 4 H.as_permutation_group()

/home/vit/bin/sage-dev/local/lib/python2.7/site-packages/sage/groups/finitely_presented.pyc in as_permutation_group(self, limit)
1008                 coset_table = self.gap().CosetTable(trivial_subgroup).sage()
1009             except ValueError:
-> 1010                 raise ValueError('Coset enumeration exceeded limit, is the group finite?')
1011         from sage.combinat.permutation import Permutation
1012         from sage.groups.perm_gps.permgroup import PermutationGroup

ValueError: Coset enumeration exceeded limit, is the group finite?

```

But checking for equality in a finitely presented group can still lead to the error described here.

```k = 3
p = 18
d = 2*p/(p-6)
l = 2*p*k/(p*k-2*k-2*p)
F.<J,P,R1,R2>=FreeGroup()
G = F / [J^3, P^(3*d), R1^p, R2^p, (F([-2,1]))^k, (F([4,3,1]))^l, F([-4,2,3,-2]), F([-4,1,3,-1]), F([-2,3,4])]
k = G.rewriting_system()
x = k.reduce(P^8)
y = k.reduce(G([-4,2,3]))
t = k.reduce(x*y)
t == G.one()
```

### comment:7 Changed 2 years ago by dimpase

Checking for equality is undesideable, GAP does try coset enumeration. I suppose you are saying Sage should recover gracefully from this. It is hard as GAP manages memory itself, and crashes.

### comment:8 Changed 2 years ago by vittucek

Yes, that's what I am saying. Whatever GAP does it shouldn't take Sage with it.

### comment:9 in reply to: ↑ 6 Changed 2 years ago by jdemeyer

the error described here.

Details please. Can you update the ticket description?

### comment:10 Changed 2 years ago by vittucek

• Description modified (diff)
• Milestone changed from sage-6.4 to sage-8.2
• Summary changed from LibGAP exits when out of memory to LibGAP exits when out of memory and takes Sage with it

### comment:11 follow-up: ↓ 12 Changed 2 years ago by jdemeyer

This ticket is something that should be fixed after the GAP + libGAP convergence. Once GAP == libGAP has a proper error handler, this should be easy to fix.

### comment:12 in reply to: ↑ 11 Changed 2 years ago by slelievre

• Dependencies set to #22626
• Description modified (diff)

This ticket is something that should be fixed after the GAP + libGAP convergence. Once GAP == libGAP has a proper error handler, this should be easy to fix.

Marking #22626 as a dependency to reflect that.

### comment:13 Changed 11 months ago by embray

This is likely OBE but it would be good to double-check what happens in this case with the new implementation.

### comment:14 Changed 11 months ago by embray

It seems that I get a segmentation fault, which is to be expected. But at least it's caught by cysignals and raised as a Python exception and everything seems to recover fine. So I think that's what we want here? I'm not sure that there's a better solution.

### comment:15 Changed 11 months ago by embray

I guess at the very least the Sage code for this equality comparison should catch the exception and do something more user-friendly with it.

Note: See TracTickets for help on using tickets.