Sage: Ticket #28106: doctest memlimit being hit in sage.combinat.tableau tests
https://trac.sagemath.org/ticket/28106
<p>
This is a follow-up to <a class="closed ticket" href="https://trac.sagemath.org/ticket/27681" title="#27681: defect: GAP: Apparent problem with workspace initialization (closed: fixed)">#27681</a> which is now closed, as it fixed one issue related to libgap and pexpect gap possibly interacting in bad ways.
</p>
<p>
However, the original issue, which for a while I could not reproduce, seems to be coming up again. For example if I run just:
</p>
<pre class="wiki">$ ./sage -t --long src/sage/combinat/tableau.py
</pre><p>
I get
</p>
<pre class="wiki">too many failed tests, not using stored timings
Running doctests with ID 2019-07-03-13-48-33-78ca2800.
Git branch: u/embray/docker-python3
Using --optional=build,dochtml,memlimit,mpir,notedown,pandoc_attributes,python2,rst2ipynb,sage
Doctesting 1 file.
sage -t --long src/sage/combinat/tableau.py
**********************************************************************
File "src/sage/combinat/tableau.py", line 7751, in sage.combinat.tableau.StandardTableaux_size.cardinality
Failed example:
StandardTableaux(50).cardinality() # long time
Exception raised:
Traceback (most recent call last):
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1105, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.combinat.tableau.StandardTableaux_size.cardinality[4]>", line 1, in <module>
StandardTableaux(Integer(50)).cardinality() # long time
File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3691)
return self.get_object()(*args, **kwds)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
return cls.classcall(cls, *args, **kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7584, in __classcall_private__
return StandardTableaux_size(n)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
return cls.classcall(cls, *args, **kwds)
File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
w = self.f(*args, **kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
instance = typecall(cls, *args, **options)
File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
return (<PyTypeObject*>type).tp_call(cls, args, kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7693, in __init__
facade=True, keepkey=False)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/disjoint_union_enumerated_sets.py", line 288, in __init__
self._facade_for = tuple(family)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1062, in __iter__
yield self[i]
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1078, in __getitem__
return self.function(i)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
return cls.classcall(cls, *args, **kwds)
File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
w = self.f(*args, **kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
instance = typecall(cls, *args, **options)
File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
return (<PyTypeObject*>type).tp_call(cls, args, kwds)
MemoryError
**********************************************************************
File "src/sage/combinat/tableau.py", line 8572, in sage.combinat.tableau.IncreasingTableaux.__init__
Failed example:
TestSuite(S).run() # long time
Exception raised:
Traceback (most recent call last):
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1118, in compile_and_execute
self.update_digests(example)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1009, in update_digests
digest.update(str_to_bytes(reduce_hex(gen), 'ascii'))
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/parsing.py", line 442, in reduce_hex
from operator import xor
MemoryError
Process DocTestWorker-1:
Traceback (most recent call last):
File "/home/embray/src/sagemath/sage/local/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
self.run()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2149, in run
task(self.options, self.outtmpfile, msgpipe, self.result_queue)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2508, in __call__
result_queue.put(result, False)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/multiprocessing/queues.py", line 107, in put
self._start_thread()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/multiprocessing/queues.py", line 195, in _start_thread
self._thread.start()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
Bad exit: 1
</pre><p>
There is something about <code>sage.combinat.tableau</code> that the problem is always occurring there, though I'm not sure why. It's usually crashing on the same tests, but not always with the exact same traceback, suggesting some pseudo-deterministic memory corruption.
</p>
en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/28106
Trac 1.2Erik BrayWed, 03 Jul 2019 14:09:14 GMT
https://trac.sagemath.org/ticket/28106#comment:1
https://trac.sagemath.org/ticket/28106#comment:1
<p>
On <a class="closed ticket" href="https://trac.sagemath.org/ticket/27681" title="#27681: defect: GAP: Apparent problem with workspace initialization (closed: fixed)">#27681</a> it was reported that the error doesn't occur when running the tests with <code>--serial</code>, but my experience is different. I still get:
</p>
<pre class="wiki">$ ./sage -t --serial --long src/sage/combinat/tableau.py
too many failed tests, not using stored timings
Running doctests with ID 2019-07-03-13-55-40-8a0bf5f9.
Git branch: u/embray/docker-python3
Using --optional=build,dochtml,memlimit,mpir,notedown,pandoc_attributes,python2,rst2ipynb,sage
Doctesting 1 file.
sage -t --long src/sage/combinat/tableau.py
**********************************************************************
File "src/sage/combinat/tableau.py", line 7751, in sage.combinat.tableau.StandardTableaux_size.cardinality
Failed example:
StandardTableaux(50).cardinality() # long time
Exception raised:
Traceback (most recent call last):
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1105, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.combinat.tableau.StandardTableaux_size.cardinality[4]>", line 1, in <module>
StandardTableaux(Integer(50)).cardinality() # long time
File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3691)
return self.get_object()(*args, **kwds)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
return cls.classcall(cls, *args, **kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7584, in __classcall_private__
return StandardTableaux_size(n)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
return cls.classcall(cls, *args, **kwds)
File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
w = self.f(*args, **kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
instance = typecall(cls, *args, **options)
File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
return (<PyTypeObject*>type).tp_call(cls, args, kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7693, in __init__
facade=True, keepkey=False)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/disjoint_union_enumerated_sets.py", line 288, in __init__
self._facade_for = tuple(family)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1062, in __iter__
yield self[i]
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1078, in __getitem__
return self.function(i)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
return cls.classcall(cls, *args, **kwds)
File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
w = self.f(*args, **kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
instance = typecall(cls, *args, **options)
File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
return (<PyTypeObject*>type).tp_call(cls, args, kwds)
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7869, in __init__
super(StandardTableaux_shape, self).__init__(category=FiniteEnumeratedSets())
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 6052, in __init__
self.max_entry = None
MemoryError
----------------------------------------------------------------------
Doctests interrupted: 0/1 files tested
----------------------------------------------------------------------
Total time for all tests: 59.6 seconds
cpu time: 0.0 seconds
cumulative wall time: 0.0 seconds
Traceback (most recent call last):
File "/home/embray/src/sagemath/sage/src/bin/sage-runtests", line 179, in <module>
err = DC.run()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 1232, in run
self.run_doctests()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 933, in run_doctests
self.dispatcher.dispatch()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2008, in dispatch
self.serial_dispatch()
File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1671, in serial_dispatch
output = bytes_to_str(outtmpfile.read())
MemoryError
</pre><p>
Since this is a pretty random issue, it possible that running with <code>--serial</code> just happened to make it go away in their case.
</p>
TicketErik BrayWed, 03 Jul 2019 14:24:45 GMT
https://trac.sagemath.org/ticket/28106#comment:2
https://trac.sagemath.org/ticket/28106#comment:2
<p>
At this point I'm not even certain it has anything to with libgap. That's just the context in which this first started appearing for me. It could also be a CPython bug.
</p>
TicketErik BrayWed, 03 Jul 2019 14:43:47 GMT
https://trac.sagemath.org/ticket/28106#comment:3
https://trac.sagemath.org/ticket/28106#comment:3
<p>
Sometimes I forget that on Linux we run the tests by default with <code>--memlimit=3300</code> which sets <code>RLIMIT_AS</code>.
</p>
<p>
I wonder if that's all it is.
</p>
TicketErik BrayWed, 03 Jul 2019 15:16:58 GMTsummary changed
https://trac.sagemath.org/ticket/28106#comment:4
https://trac.sagemath.org/ticket/28106#comment:4
<ul>
<li><strong>summary</strong>
changed from <em>Memory corruption possibly related to libgap</em> to <em>doctest memlimit being hit in sage.combinat.tableau tests</em>
</li>
</ul>
<p>
Yep, the crash is just occurring upon hitting the 3300MB memlimit. Simple and mundane as that.
</p>
<p>
It could still be indirectly related to libgap. When investigating this earlier in <a class="closed ticket" href="https://trac.sagemath.org/ticket/27681" title="#27681: defect: GAP: Apparent problem with workspace initialization (closed: fixed)">#27681</a> we thought it might have something to do with GAP workspaces. This is because I first discovered the problem while working on <a class="closed ticket" href="https://trac.sagemath.org/ticket/18267" title="#18267: enhancement: libgap for PermutationGroup (closed: fixed)">#18267</a>, in which libgap is used much more extensively directly in sage (as opposed to pexpect gap which is running outside the main sage process and so doesn't hit this memory limit as easily). It also seemed to go away if I deleted my existing GAP workspaces.
</p>
<p>
It could be that in the previous case, merely loading a pre-existing GAP workspace was enough to push over the memory limit.
</p>
TicketErik BrayWed, 03 Jul 2019 15:56:05 GMT
https://trac.sagemath.org/ticket/28106#comment:5
https://trac.sagemath.org/ticket/28106#comment:5
<p>
Of note: With the default of <code>--memlimit=3300</code>, roughly 2GB of virtual memory are already reserved by various things just while importing Sage. Of that, about 250MB is used just by Python and various loaded modules. 825 MB (i.e. <code>virtual_memory_limit() // 4</code>) are reserved by Pari, which is loaded no matter what. The rest I can't easily account for yet...
</p>
<p>
While running the tests, libgap reserves initially what amounts to <code>virtual_memory_limit() // 10</code>, or only 330 MB, but the 2 GB I mentioned previously doesn't even include libgap.
</p>
<p>
I wonder, if nothing else, as long as we're capping the virtual memory limit for doctests, the default amount reserved for Pari should be lower.
</p>
TicketErik BrayThu, 04 Jul 2019 14:39:25 GMTstatus changed; branch, author set
https://trac.sagemath.org/ticket/28106#comment:6
https://trac.sagemath.org/ticket/28106#comment:6
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_review</em>
</li>
<li><strong>branch</strong>
set to <em>u/embray/ticket-28106</em>
</li>
<li><strong>author</strong>
set to <em>Erik Bray</em>
</li>
</ul>
<p>
One little thing we can do that happens to be good-enough for this case, it seems, is to further limit the amount of workspace Pari allocates by default, just when running the test suite.
</p>
<p>
Since many tests won't require that much memory for Pari this seems sensible.
</p>
<p>
In the case of this particular test the other issue is just reducing the amount of memory used when instantiating large Tableaux classes. I have opened <a class="new ticket" href="https://trac.sagemath.org/ticket/28114" title="#28114: enhancement: Back tableaux by Numpy arrays where possible (new)">#28114</a> for that.
</p>
TicketgitThu, 04 Jul 2019 14:39:52 GMTcommit set
https://trac.sagemath.org/ticket/28106#comment:7
https://trac.sagemath.org/ticket/28106#comment:7
<ul>
<li><strong>commit</strong>
set to <em>52b922ff9310b8ef5cda788e11a4cb7be5fe4895</em>
</li>
</ul>
<p>
Branch pushed to git repo; I updated commit sha1. New commits:
</p>
<table class="wiki">
<tr><td><a class="ext-link" href="https://git.sagemath.org/sage.git/commit/?id=52b922ff9310b8ef5cda788e11a4cb7be5fe4895"><span class="icon"></span>52b922f</a></td><td><code>Ticket #28106: Limit the initial memory allocation for Pari even further when running tests.</code>
</td></tr></table>
TicketTravis ScrimshawMon, 08 Jul 2019 13:20:09 GMT
https://trac.sagemath.org/ticket/28106#comment:8
https://trac.sagemath.org/ticket/28106#comment:8
<p>
I am a little split on this. I agree that we don't really need it for the doctests, but I don't like making the behavior different than what is done within a Sage session. Or is there something special with Cygwin and/or the doctest framework going on here?
</p>
TicketMarkus WageringelThu, 11 Jul 2019 20:19:00 GMT
https://trac.sagemath.org/ticket/28106#comment:9
https://trac.sagemath.org/ticket/28106#comment:9
<p>
This problem can also be observed with a doctest for <code>divisors()</code> in <code>sage/rings/integer.pyx</code>. See <a class="closed ticket" href="https://trac.sagemath.org/ticket/28162" title="#28162: defect: AlarmInterrupt in divisors doctest in integer.pyx (closed: duplicate)">#28162</a>.
</p>
<p>
I tried the branch here, but it does not solve the issue for me. Both the tests in <code>tableau.py</code> and <code>integer.pyx</code> still fail.
</p>
<p>
On the other hand, raising the <code>memlimit</code> from 3300 to 3400 makes the tests pass for both of the files, without this branch even. If I understand the change correctly, it should decrease memory usage by much more than 100 MB, so I am not sure what is going on.
</p>
TicketVolker BraunSat, 13 Jul 2019 09:40:36 GMTstatus changed; reviewer set
https://trac.sagemath.org/ticket/28106#comment:10
https://trac.sagemath.org/ticket/28106#comment:10
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
set to <em>Volker Braun</em>
</li>
</ul>
<p>
I'm also hitting this on the buildbot, unsurprisingly.
</p>
<p>
IMHO its reasonable to provide less memory specifically for doctests, if only to encourage authors to limit resource usage when writing tests.
</p>
TicketVolker BraunSat, 13 Jul 2019 11:57:33 GMTstatus changed
https://trac.sagemath.org/ticket/28106#comment:11
https://trac.sagemath.org/ticket/28106#comment:11
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>needs_work</em>
</li>
</ul>
<p>
I'm still running sometimes out of memory
</p>
<pre class="wiki">Using --optional=4ti2,benzene,buckygen,build,cmake,database_cremona_ellcurve,database_jones_numfield,database_kohel,database_mutation_class,database_odlyzko_zeta,database_stein_watkins,database_stein_watkins_mini,database_symbolic_data,deformation,dochtml,e_antic,gambit,latte_int,libogg,lidia,lrslib,memlimit,mpir,normaliz,ore_algebra,pari_elldata,pari_galpol,pari_nftables,pari_seadata,plantri,python2,qhull,saclib,sage,tides,topcom
Doctesting entire Sage library.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 3816 files using 16 threads.
**********************************************************************
File "src/sage/combinat/tableau.py", line 7698, in sage.combinat.tableau.StandardTableaux_size.cardinality
Failed example:
StandardTableaux(50).cardinality() # long time
Exception raised:
Traceback (most recent call last):
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
self.compile_and_execute(example, compiler, test.globs)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1105, in compile_and_execute
exec(compiled, globs)
File "<doctest sage.combinat.tableau.StandardTableaux_size.cardinality[4]>", line 1, in <module>
StandardTableaux(Integer(50)).cardinality() # long time
File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3697)
return self.get_object()(*args, **kwds)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1741)
return cls.classcall(cls, *args, **kwds)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7531, in __classcall_private__
return StandardTableaux_size(n)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1741)
return cls.classcall(cls, *args, **kwds)
File "sage/misc/cachefunc.pyx", line 1002, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6017)
w = self.f(*args, **kwds)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
instance = typecall(cls, *args, **options)
File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2191)
return (<PyTypeObject*>type).tp_call(cls, args, kwds)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7640, in __init__
facade=True, keepkey=False)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/sets/disjoint_union_enumerated_sets.py", line 288, in __init__
self._facade_for = tuple(family)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/sets/family.py", line 1062, in __iter__
yield self[i]
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/sets/family.py", line 1078, in __getitem__
return self.function(i)
File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1741)
return cls.classcall(cls, *args, **kwds)
File "sage/misc/cachefunc.pyx", line 1002, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6017)
w = self.f(*args, **kwds)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
instance = typecall(cls, *args, **options)
File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2191)
return (<PyTypeObject*>type).tp_call(cls, args, kwds)
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7816, in __init__
super(StandardTableaux_shape, self).__init__(category=FiniteEnumeratedSets())
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 6000, in __init__
Tableaux.__init__(self, **kwds)
File "sage/structure/parent.pyx", line 296, in sage.structure.parent.Parent.__init__ (build/cythonized/sage/structure/parent.c:4589)
CategoryObject.__init__(self, category, base)
File "sage/structure/category_object.pyx", line 115, in sage.structure.category_object.CategoryObject.__init__ (build/cythonized/sage/structure/category_object.c:2053)
self._init_category_(category)
File "sage/structure/parent.pyx", line 339, in sage.structure.parent.Parent._init_category_ (build/cythonized/sage/structure/parent.c:5059)
self.__class__ = dynamic_class(
File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/structure/dynamic_class.py", line 335, in dynamic_class
return dynamic_class_internal(name, bases, cls, reduction, doccls, prepend_cls_bases)
MemoryError
**********************************************************************
</pre>
TicketDima PasechnikSat, 13 Jul 2019 12:09:06 GMT
https://trac.sagemath.org/ticket/28106#comment:12
https://trac.sagemath.org/ticket/28106#comment:12
<p>
I still see this on my Fedora desktop, as I run tests with <code>--long</code> on. There seems to be something Fedora-specific here.
</p>
TicketVolker BraunSat, 13 Jul 2019 12:12:47 GMT
https://trac.sagemath.org/ticket/28106#comment:13
https://trac.sagemath.org/ticket/28106#comment:13
<p>
Can we just reduce <code>StandardTableaux(50)</code> to something smaller here?
</p>
<p>
This is on Debian fwiw.
</p>
TicketDima PasechnikSat, 13 Jul 2019 12:31:04 GMT
https://trac.sagemath.org/ticket/28106#comment:14
https://trac.sagemath.org/ticket/28106#comment:14
<p>
hmm, but it is fast (for me, on Debian 10):
</p>
<pre class="wiki">sage: timeit("StandardTableaux(50).cardinality()")
5 loops, best of 3: 169 μs per loop
</pre><p>
It is not constructing this huge set for sure...
</p>
TicketTravis ScrimshawSat, 13 Jul 2019 13:35:06 GMT
https://trac.sagemath.org/ticket/28106#comment:15
https://trac.sagemath.org/ticket/28106#comment:15
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:14" title="Comment 14">dimpase</a>:
</p>
<blockquote class="citation">
<p>
hmm, but it is fast (for me, on Debian 10):
</p>
<pre class="wiki">sage: timeit("StandardTableaux(50).cardinality()")
5 loops, best of 3: 169 μs per loop
</pre><p>
It is not constructing this huge set for sure...
</p>
</blockquote>
<p>
Which is the point of this doctest, the set is huge and it is using an actual computation. However, <code>StandardPartitions</code> does need to compute the partitions of 50 as part of the <code>__init__</code> as part of this line:
</p>
<pre class="wiki">Family(Partitions_n(n), StandardTableaux_shape),
</pre><p>
Now the <code>Family</code> is created lazily; however, this gets expanded by the call <code>self._facade_for = tuple(family)</code> in the <code>DisjointUnionEnumeratedSets.__init__</code>. So we construct a set of size
</p>
<pre class="wiki">sage: Partitions(50).cardinality()
204226
</pre><p>
instead of
</p>
<pre class="wiki">sage: ST = StandardTableaux(50)
sage: ST.cardinality()
27886995605342342839104615869259776
</pre><p>
Now unfortunately I do not see a simple way to keep that family done lazily. However, changing it to 40 should still do an appropriate test but uses far less memory:
</p>
<pre class="wiki">sage: Partitions(40).cardinality()
37338
sage: StandardTableaux(40).cardinality()
72682301192087742711233536
</pre>
TicketMarkus WageringelWed, 24 Jul 2019 09:31:04 GMT
https://trac.sagemath.org/ticket/28106#comment:16
https://trac.sagemath.org/ticket/28106#comment:16
<p>
The current branch does not work as intended because <code>DOCTEST_MODE</code> is not set yet when Pari is initialized. Perhaps it could be set earlier on, in sage-runtests?
</p>
TicketMarkus WageringelWed, 24 Jul 2019 14:49:04 GMT
https://trac.sagemath.org/ticket/28106#comment:17
https://trac.sagemath.org/ticket/28106#comment:17
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:15" title="Comment 15">tscrim</a>:
</p>
<blockquote class="citation">
<p>
Now unfortunately I do not see a simple way to keep that family done lazily.
</p>
</blockquote>
<p>
The following seems to work. All tests pass for me.
</p>
<div class="wiki-code"><div xmlns="http://www.w3.org/1999/xhtml" class="diff">
<ul class="entries">
<li class="entry">
<h2>
<a>src/sage/categories/facade_sets.py</a>
</h2>
<table class="trac-diff inline" summary="Differences" cellspacing="0">
<colgroup><col class="lineno" /><col class="lineno" /><col class="content" /></colgroup>
<thead>
<tr>
<th title="File a/src/sage/categories/facade_sets.py">
a
</th>
<th title="File b/src/sage/categories/facade_sets.py">
b
</th>
<td><em> class FacadeSets(CategoryWithAxiom):</em> </td>
</tr>
</thead>
<tbody class="unmod">
<tr>
<th>177</th><th>177</th><td class="l"><span> if parents is True:</span></td>
</tr><tr>
<th>178</th><th>178</th><td class="l"><span> return True</span></td>
</tr><tr>
<th>179</th><th>179</th><td class="l"><span> from sage.structure.element import parent</span></td>
</tr>
</tbody><tbody class="mod">
<tr class="first">
<th>180</th><th> </th><td class="l"><span> return parent(element) in parents</span></td>
</tr>
<tr>
<th> </th><th>180</th><td class="r"><span> # parents is a tuple or (finite lazy) family of Parent types;</span></td>
</tr><tr>
<th> </th><th>181</th><td class="r"><span> # LazyFamily does not implement __contains__, so we iterate over it</span></td>
</tr><tr class="last">
<th> </th><th>182</th><td class="r"><span> return parent(element) in iter(parents)</span></td>
</tr>
</tbody><tbody class="unmod">
<tr>
<th>181</th><th>183</th><td class="l"><span></span></td>
</tr><tr>
<th>182</th><th>184</th><td class="l"><span> def __contains__(self, element):</span></td>
</tr><tr>
<th>183</th><th>185</th><td class="l"><span> """</span></td>
</tr>
</tbody>
</table>
</li>
<li class="entry">
<h2>
<a>src/sage/sets/disjoint_union_enumerated_sets.py</a>
</h2>
<table class="trac-diff inline" summary="Differences" cellspacing="0">
<colgroup><col class="lineno" /><col class="lineno" /><col class="content" /></colgroup>
<thead>
<tr>
<th title="File a/src/sage/sets/disjoint_union_enumerated_sets.py">
a
</th>
<th title="File b/src/sage/sets/disjoint_union_enumerated_sets.py">
b
</th>
<td><em> class DisjointUnionEnumeratedSets(UniqueRepresentation, Parent):</em> </td>
</tr>
</thead>
<tbody class="unmod">
<tr>
<th>285</th><th>285</th><td class="l"><span> self._facade = facade</span></td>
</tr><tr>
<th>286</th><th>286</th><td class="l"><span> if facade:</span></td>
</tr><tr>
<th>287</th><th>287</th><td class="l"><span> if family in FiniteEnumeratedSets():</span></td>
</tr>
</tbody><tbody class="mod">
<tr class="first">
<th>288</th><th> </th><td class="l"><span> self._facade_for = <del>tuple(family)</del></span></td>
</tr>
<tr class="last">
<th> </th><th>288</th><td class="r"><span> self._facade_for = <ins>family</ins></span></td>
</tr>
</tbody><tbody class="unmod">
<tr>
<th>289</th><th>289</th><td class="l"><span> else:</span></td>
</tr><tr>
<th>290</th><th>290</th><td class="l"><span> # This allows the test suite to pass its tests by essentially</span></td>
</tr><tr>
<th>291</th><th>291</th><td class="l"><span> # stating that this is a facade for any parent. Technically</span></td>
</tr>
</tbody>
<tbody class="skipped">
<tr>
<th><a href="#L576">…</a></th>
<th><a href="#L576">…</a></th>
<td><em> class DisjointUnionEnumeratedSets(UniqueRepresentation, Parent):</em> </td>
</tr>
</tbody>
<tbody class="unmod">
<tr>
<th>576</th><th>576</th><td class="l"><span> raise ValueError("cannot coerce `%s` in the parent `%s`"%(el[1], P))</span></td>
</tr><tr>
<th>577</th><th>577</th><td class="l"><span></span></td>
</tr><tr>
<th>578</th><th>578</th><td class="l"><span> # Check first to see if the parent of el is in the family</span></td>
</tr>
</tbody><tbody class="mod">
<tr class="first">
<th>579</th><th> </th><td class="l"><span> if (isinstance(el, Element) and <del>isinstance(self._facade_for, tuple)</del></span></td>
</tr><tr>
<th>580</th><th> </th><td class="l"><span> and el.parent() in <del>self._facade_for</del>):</span></td>
</tr>
<tr>
<th> </th><th>579</th><td class="r"><span> if (isinstance(el, Element) and <ins>self._facade_for is not True</ins></span></td>
</tr><tr class="last">
<th> </th><th>580</th><td class="r"><span> and el.parent() in <ins>iter(self._facade_for)</ins>):</span></td>
</tr>
</tbody><tbody class="unmod">
<tr>
<th>581</th><th>581</th><td class="l"><span> return el</span></td>
</tr><tr>
<th>582</th><th>582</th><td class="l"><span></span></td>
</tr><tr>
<th>583</th><th>583</th><td class="l"><span> for P in self._family:</span></td>
</tr>
</tbody>
</table>
</li>
</ul>
</div></div><p>
This change does not have the potential to solve the memory problems in integer.pyx, but it could be useful regardless of the memory issues. With this we could even test for <code>StandardTableaux(500).cardinality()</code> instead, which only takes a few milliseconds.
</p>
TicketTravis ScrimshawFri, 26 Jul 2019 20:23:53 GMT
https://trac.sagemath.org/ticket/28106#comment:18
https://trac.sagemath.org/ticket/28106#comment:18
<p>
Hmm...I thought I had some reason for not doing that when I looked at it for <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:15" title="Comment 15">comment:15</a>, but the logic all seems to checkout. There is one small behavior change: If it was given a finite iterator before, it would know it is a finite enumerated set, but now that doesn't happen (infinite iterators would loop forever before)
</p>
<pre class="wiki">sage: iter([1,2,3,4,5]) in FiniteEnumeratedSets()
False
</pre><p>
So I think that is a safe change.
</p>
TicketMarkus WageringelSat, 27 Jul 2019 14:53:52 GMT
https://trac.sagemath.org/ticket/28106#comment:19
https://trac.sagemath.org/ticket/28106#comment:19
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:18" title="Comment 18">tscrim</a>:
</p>
<blockquote class="citation">
<p>
There is one small behavior change: If it was given a finite iterator before, it would know it is a finite enumerated set, but now that doesn't happen (infinite iterators would loop forever before)
</p>
</blockquote>
<p>
I am not sure what you mean by the change in behavior. Could you give an example? The variable <code>family</code> is always a <code>Family</code> here, not an iterator, which is asserted by the <code>__classcall_private__</code> in <code>DisjointUnionEnumeratedSets</code>.
</p>
<p>
However on second thought, instead of calling <code>iter</code>, it would be clearer to implement <code>__contains__</code> for lazy families, which could then choose an appropriate behavior depending on whether the family is finite or not. When given a different kind of family, <code>__contains__</code> could potentially do something more efficient than iterating. Currently, all other subclasses of <code>Family</code> already implement it, except for <code>LazyFamily</code>. This has the effect that the default implementation from <code>Parent</code> is called for lazy families, producing a very uninformative error message.
</p>
<p>
I could prepare a branch for this. Should I create a new ticket for that or put it here?
</p>
TicketTravis ScrimshawSat, 27 Jul 2019 20:14:11 GMT
https://trac.sagemath.org/ticket/28106#comment:20
https://trac.sagemath.org/ticket/28106#comment:20
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:19" title="Comment 19">gh-mwageringel</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:18" title="Comment 18">tscrim</a>:
</p>
<blockquote class="citation">
<p>
There is one small behavior change: If it was given a finite iterator before, it would know it is a finite enumerated set, but now that doesn't happen (infinite iterators would loop forever before)
</p>
</blockquote>
<p>
I am not sure what you mean by the change in behavior. Could you give an example? The variable <code>family</code> is always a <code>Family</code> here, not an iterator, which is asserted by the <code>__classcall_private__</code> in <code>DisjointUnionEnumeratedSets</code>.
</p>
</blockquote>
<p>
I mean the same input will result in a different object being created (i.e., the behavior) with your branch if it was given an iterator/generator. You are not just changing things that affect only <code>DisjointUnionEnumeratedSets</code>. The <code>family</code> change in <code>facade_sets</code> can affect many other things. Granted, doing things this way is arguably safer, but I am just pointing out a difference here.
</p>
<blockquote class="citation">
<p>
However on second thought, instead of calling <code>iter</code>, it would be clearer to implement <code>__contains__</code> for lazy families, which could then choose an appropriate behavior depending on whether the family is finite or not. When given a different kind of family, <code>__contains__</code> could potentially do something more efficient than iterating. Currently, all other subclasses of <code>Family</code> already implement it, except for <code>LazyFamily</code>. This has the effect that the default implementation from <code>Parent</code> is called for lazy families, producing a very uninformative error message.
</p>
<p>
I could prepare a branch for this. Should I create a new ticket for that or put it here?
</p>
</blockquote>
<p>
Please do so. Be aware, a lazy family may not know if it finite or not (a priori, I haven't checked this is actually true in the code). So that part might end up being more of a can of worms, but I don't know until someone actually does it.
</p>
TicketMarkus WageringelSun, 28 Jul 2019 13:20:43 GMT
https://trac.sagemath.org/ticket/28106#comment:21
https://trac.sagemath.org/ticket/28106#comment:21
<p>
I opened <a class="closed ticket" href="https://trac.sagemath.org/ticket/28273" title="#28273: enhancement: allow laziness in DisjointUnionEnumeratedSet (closed: fixed)">#28273</a> implementing this. I created a new ticket for this, as Erik's approach tried here is justified in its own right.
</p>
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:20" title="Comment 20">tscrim</a>:
</p>
<blockquote class="citation">
<p>
I mean the same input will result in a different object being created (i.e., the behavior) with your branch if it was given an iterator/generator. You are not just changing things that affect only <code>DisjointUnionEnumeratedSets</code>. The <code>family</code> change in <code>facade_sets</code> can affect many other things. Granted, doing things this way is arguably safer, but I am just pointing out a difference here.
</p>
</blockquote>
<p>
Ok, I see. Indeed the change in <code>facade_sets</code> could affect other things – though within the Sage library it should not affect anything, as <code>_facade_for</code> is always a tuple, currently. This assumption could easily break in the future though, so I think it is safer to do things as in <a class="closed ticket" href="https://trac.sagemath.org/ticket/28273" title="#28273: enhancement: allow laziness in DisjointUnionEnumeratedSet (closed: fixed)">#28273</a>, which does not touch facade_sets.py at all.
</p>
TicketErik BrayFri, 02 Aug 2019 09:58:12 GMT
https://trac.sagemath.org/ticket/28106#comment:22
https://trac.sagemath.org/ticket/28106#comment:22
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:8" title="Comment 8">tscrim</a>:
</p>
<blockquote class="citation">
<p>
I am a little split on this. I agree that we don't really need it for the doctests, but I don't like making the behavior different than what is done within a Sage session. Or is there something special with Cygwin and/or the doctest framework going on here?
</p>
</blockquote>
<p>
Yes, the fact that the doctest framework, by default, puts a low cap on virtual memory allocation allowed by the process, which would not normally be the case unless the user is manually setting those limits.
</p>
<p>
Therefore the large vmem allocation normally allowed for PARI's initial stack needs to be scaled back that much more when running the doctests in order to leave room for other things.
</p>
TicketErik BrayFri, 02 Aug 2019 10:47:09 GMT
https://trac.sagemath.org/ticket/28106#comment:23
https://trac.sagemath.org/ticket/28106#comment:23
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:16" title="Comment 16">gh-mwageringel</a>:
</p>
<blockquote class="citation">
<p>
The current branch does not work as intended because <code>DOCTEST_MODE</code> is not set yet when Pari is initialized. Perhaps it could be set earlier on, in sage-runtests?
</p>
</blockquote>
<p>
I see, that is true. It's been several weeks since I last looked at this, but this change seemed to work for me, but now I can't be sure why. I must have been doing something differently at the time.
</p>
TicketgitFri, 02 Aug 2019 12:56:20 GMTcommit changed
https://trac.sagemath.org/ticket/28106#comment:24
https://trac.sagemath.org/ticket/28106#comment:24
<ul>
<li><strong>commit</strong>
changed from <em>52b922ff9310b8ef5cda788e11a4cb7be5fe4895</em> to <em>c6ff771f9f11e131fc55e3db6e17b778c67407d6</em>
</li>
</ul>
<p>
Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:
</p>
<table class="wiki">
<tr><td><a class="ext-link" href="https://git.sagemath.org/sage.git/commit/?id=65b276cc516cd508d38528036d69321e50e5bb6d"><span class="icon"></span>65b276c</a></td><td><code>Ticket #28106: Limit the initial memory allocation for Pari even further when running tests.</code>
</td></tr><tr><td><a class="ext-link" href="https://git.sagemath.org/sage.git/commit/?id=c6ff771f9f11e131fc55e3db6e17b778c67407d6"><span class="icon"></span>c6ff771</a></td><td><code>Fix accidental initialization of Pari during sage.doctest.parsing import</code>
</td></tr></table>
TicketErik BrayFri, 02 Aug 2019 12:57:41 GMTstatus changed
https://trac.sagemath.org/ticket/28106#comment:25
https://trac.sagemath.org/ticket/28106#comment:25
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
</ul>
<p>
I rebased the branch on current develop, and added a fix to ensure that pari was not being initialized so much earlier than intended during import of the doctest sub-package.
</p>
TicketMarkus WageringelMon, 05 Aug 2019 18:39:10 GMTstatus changed
https://trac.sagemath.org/ticket/28106#comment:26
https://trac.sagemath.org/ticket/28106#comment:26
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>needs_work</em>
</li>
</ul>
<p>
Now there are several doctest failures as a result of the reduced Pari stack size, see the patchbot, so I am setting back to needs work. I have only had a brief look, but some of them seem hard to solve.
</p>
<p>
Also, considering that relying on a specific load order could be a bit fragile, would it be possible to add a doctest or an assertion to check that Pari is loaded after <code>DOCTEST_MODE</code> is set?
</p>
TicketErik BrayTue, 06 Aug 2019 12:53:35 GMT
https://trac.sagemath.org/ticket/28106#comment:27
https://trac.sagemath.org/ticket/28106#comment:27
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:26" title="Comment 26">gh-mwageringel</a>:
</p>
<blockquote class="citation">
<p>
Also, considering that relying on a specific load order could be a bit fragile, would it be possible to add a doctest or an assertion to check that Pari is loaded after <code>DOCTEST_MODE</code> is set?
</p>
</blockquote>
<p>
I thought about that but I'm not sure exactly a good way to do that. It might be necessary to add some kind of pari_initialized flag.
</p>
TicketErik BrayTue, 06 Aug 2019 13:01:03 GMT
https://trac.sagemath.org/ticket/28106#comment:28
https://trac.sagemath.org/ticket/28106#comment:28
<p>
I wonder what the absolute minimum PARI stack size needed is. We shouldn't go below that, sure, but it's not really clear what the limit is here...
</p>
TicketDima PasechnikTue, 06 Aug 2019 13:43:15 GMT
https://trac.sagemath.org/ticket/28106#comment:29
https://trac.sagemath.org/ticket/28106#comment:29
<p>
Pari initialisation has 2 parameters, one is stack size, another some kind of “minimal prime”. I had to deal with it my recent branch on <a class="closed ticket" href="https://trac.sagemath.org/ticket/28242" title="#28242: enhancement: spkg-configure.m4 for pari (closed: fixed)">#28242</a>...
</p>
TicketJeroen DemeyerTue, 06 Aug 2019 15:34:06 GMT
https://trac.sagemath.org/ticket/28106#comment:30
https://trac.sagemath.org/ticket/28106#comment:30
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:10" title="Comment 10">vbraun</a>:
</p>
<blockquote class="citation">
<p>
IMHO its reasonable to provide less memory specifically for doctests, if only to encourage authors to limit resource usage when writing tests.
</p>
</blockquote>
<p>
I added that limit. My main motivation was to prevent accidents where Sage doctests would trash a machine by using all memory (that happened to me multiple times). The limit of 3300 MB was simply chosen as the smallest number that made all doctests pass at the time when that limit was imposed.
</p>
TicketJeroen DemeyerTue, 06 Aug 2019 15:42:49 GMT
https://trac.sagemath.org/ticket/28106#comment:31
https://trac.sagemath.org/ticket/28106#comment:31
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:22" title="Comment 22">embray</a>:
</p>
<blockquote class="citation">
<p>
Therefore the large vmem allocation normally allowed for PARI's initial stack needs to be scaled back that much more when running the doctests in order to leave room for other things.
</p>
</blockquote>
<p>
I don't like adding yet another special case for doctests. Why not simply increase the 3300 MB memory limit a bit?
</p>
TicketJeroen DemeyerTue, 06 Aug 2019 15:50:46 GMT
https://trac.sagemath.org/ticket/28106#comment:32
https://trac.sagemath.org/ticket/28106#comment:32
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:22" title="Comment 22">embray</a>:
</p>
<blockquote class="citation">
<p>
Therefore the large vmem allocation normally allowed for PARI's initial stack needs to be scaled back that much more when running the doctests in order to leave room for other things.
</p>
</blockquote>
<p>
I wouldn't be surprised if some tests (in particular <code>heegner.py</code>) used most of the available PARI stack. So I doubt that there is much room for making the PARI stack smaller during doctests.
</p>
TicketErik BrayMon, 12 Aug 2019 11:25:07 GMT
https://trac.sagemath.org/ticket/28106#comment:33
https://trac.sagemath.org/ticket/28106#comment:33
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:30" title="Comment 30">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:10" title="Comment 10">vbraun</a>:
</p>
<blockquote class="citation">
<p>
IMHO its reasonable to provide less memory specifically for doctests, if only to encourage authors to limit resource usage when writing tests.
</p>
</blockquote>
<p>
I added that limit. My main motivation was to prevent accidents where Sage doctests would trash a machine by using all memory (that happened to me multiple times). The limit of 3300 MB was simply chosen as the smallest number that made all doctests pass at the time when that limit was imposed.
</p>
</blockquote>
<p>
It just doesn't seem "fair", in a sense, that we impose a default hard limit on virtual memory allocation, and then have a large chunk of that eaten up by Pari, even when most of it isn't needed for most other tests.
</p>
<p>
When Pari runs out of memory isn't there some way to "catch" that, increase its stack, and re-try the operation? Perhaps tests/code that are expected to eat up the Pari stack should actually explicitly do that (it would be good to test that anyways).
</p>
TicketJeroen DemeyerTue, 13 Aug 2019 09:06:52 GMT
https://trac.sagemath.org/ticket/28106#comment:34
https://trac.sagemath.org/ticket/28106#comment:34
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:33" title="Comment 33">embray</a>:
</p>
<blockquote class="citation">
<p>
It just doesn't seem "fair", in a sense, that we impose a default hard limit on virtual memory allocation, and then have a large chunk of that eaten up by Pari, even when most of it isn't needed for most other tests.
</p>
</blockquote>
<p>
I really don't understand at all what the word "fair" (even you put it in quotes) means here. This isn't about fairness, it's about solving a practical problem, namely preventing excessive memory usage by doctests. As I said: the limit of 3300 MB is chosen pretty arbitrarily and I wouldn't mind increasing it.
</p>
<blockquote class="citation">
<p>
When Pari runs out of memory isn't there some way to "catch" that, increase its stack, and re-try the operation?
</p>
</blockquote>
<p>
Sure, it probably could be done, but why bother? It's much easier to just allocate a large virtual stack. I just don't see the problem with that.
</p>
TicketVolker BraunTue, 13 Aug 2019 09:42:17 GMT
https://trac.sagemath.org/ticket/28106#comment:35
https://trac.sagemath.org/ticket/28106#comment:35
<p>
Its also the only bound on actual ram usage during tests, and we should try to keep that manageable especially since tests tend to be run in parallel and cpu core count is increasing faster than ram prices are declining. Also, something <= 4gb is going to be the upper bound on 32-bit anyways.
</p>
TicketErik BrayTue, 13 Aug 2019 10:03:28 GMT
https://trac.sagemath.org/ticket/28106#comment:36
https://trac.sagemath.org/ticket/28106#comment:36
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:34" title="Comment 34">jdemeyer</a>:
</p>
<blockquote class="citation">
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:33" title="Comment 33">embray</a>:
</p>
<blockquote class="citation">
<p>
It just doesn't seem "fair", in a sense, that we impose a default hard limit on virtual memory allocation, and then have a large chunk of that eaten up by Pari, even when most of it isn't needed for most other tests.
</p>
</blockquote>
<p>
I really don't understand at all what the word "fair" (even you put it in quotes) means here. This isn't about fairness, it's about solving a practical problem, namely preventing excessive memory usage by doctests. As I said: the limit of 3300 MB is chosen pretty arbitrarily and I wouldn't mind increasing it.
</p>
<blockquote class="citation">
<p>
When Pari runs out of memory isn't there some way to "catch" that, increase its stack, and re-try the operation?
</p>
</blockquote>
<p>
Sure, it probably could be done, but why bother? It's much easier to just allocate a large virtual stack. I just don't see the problem with that.
</p>
</blockquote>
<p>
That's what I'm talking about: Normally it's fine to just allocate a large stack for PARI. But RLIMIT_AS limits the amount of virtual address space available to the process, and a large chunk of that gets used, without exception, for the PARI stack even for tests that don't need it so much, thus creating stricter limitations on what those tests are allowed to do. This is what I mean by "fair".
</p>
TicketErik BrayWed, 04 Sep 2019 10:31:07 GMT
https://trac.sagemath.org/ticket/28106#comment:37
https://trac.sagemath.org/ticket/28106#comment:37
<p>
I've since seen various other instances of this issue, not so much on my own machine, but others seem to be encountering it in various tests. Although I can't prove myself that this is the reason (they would have to demonstrate that the failing tests are hitting the VM limit, or that the problems go away with <code>--memlimit=0</code>.
</p>
<p>
But ISTM we need to do <em>something</em> about this. If not in the PARI interface I could also see about reducing the amount libgap needs to allocate, as I believe it's the increased usage of libgap in permutation groups that is largely responsible for the increase in issues like this.
</p>
TicketJeroen DemeyerWed, 04 Sep 2019 18:49:11 GMT
https://trac.sagemath.org/ticket/28106#comment:38
https://trac.sagemath.org/ticket/28106#comment:38
<p>
Why aren't we just doing the obvious thing and increasing the limit from 3300MB to 3400MB? There is nothing magical about that 3300MB limit, it's mostly an arbitrary number.
</p>
TicketErik BrayWed, 18 Sep 2019 12:27:46 GMT
https://trac.sagemath.org/ticket/28106#comment:39
https://trac.sagemath.org/ticket/28106#comment:39
<p>
Sure, I obviously wouldn't mind increasing the limit. I just wonder how sustainable that is. Perhaps a few more tests should be marked <code># optional - memlimit</code> if this becomes a frequent problem.
</p>
<p>
It also makes me wonder if there aren't memory leaks that should be investigated.
</p>
TicketErik BrayMon, 30 Dec 2019 14:48:17 GMTmilestone changed
https://trac.sagemath.org/ticket/28106#comment:40
https://trac.sagemath.org/ticket/28106#comment:40
<ul>
<li><strong>milestone</strong>
changed from <em>sage-8.9</em> to <em>sage-9.1</em>
</li>
</ul>
<p>
Ticket retargeted after milestone closed
</p>
TicketMatthias KöppeThu, 07 May 2020 02:59:18 GMTmilestone changed
https://trac.sagemath.org/ticket/28106#comment:41
https://trac.sagemath.org/ticket/28106#comment:41
<ul>
<li><strong>milestone</strong>
changed from <em>sage-9.1</em> to <em>sage-9.2</em>
</li>
</ul>
TicketMatthias KöppeSat, 24 Oct 2020 20:15:01 GMTmilestone changed
https://trac.sagemath.org/ticket/28106#comment:42
https://trac.sagemath.org/ticket/28106#comment:42
<ul>
<li><strong>milestone</strong>
changed from <em>sage-9.2</em> to <em>sage-9.3</em>
</li>
</ul>
TicketMatthias KöppeSat, 13 Feb 2021 20:51:01 GMTmilestone changed
https://trac.sagemath.org/ticket/28106#comment:43
https://trac.sagemath.org/ticket/28106#comment:43
<ul>
<li><strong>milestone</strong>
changed from <em>sage-9.3</em> to <em>sage-9.4</em>
</li>
</ul>
<p>
Setting new milestone based on a cursory review of ticket status, priority, and last modification date.
</p>
TicketSamuel LelièvreTue, 06 Apr 2021 11:16:41 GMT
https://trac.sagemath.org/ticket/28106#comment:44
https://trac.sagemath.org/ticket/28106#comment:44
<p>
Ticket <a class="closed ticket" href="https://trac.sagemath.org/ticket/31395" title="#31395: defect: RuntimeError: can't start new thread (due to memlimit) (closed: fixed)">#31395</a> removed <code>memlimit</code> for doctests.
</p>
TicketMarkus WageringelFri, 09 Apr 2021 19:50:33 GMTmilestone changed
https://trac.sagemath.org/ticket/28106#comment:45
https://trac.sagemath.org/ticket/28106#comment:45
<ul>
<li><strong>milestone</strong>
changed from <em>sage-9.4</em> to <em>sage-duplicate/invalid/wontfix</em>
</li>
</ul>
<p>
Replying to <a class="ticket" href="https://trac.sagemath.org/ticket/28106#comment:44" title="Comment 44">slelievre</a>:
</p>
<blockquote class="citation">
<p>
Ticket <a class="closed ticket" href="https://trac.sagemath.org/ticket/31395" title="#31395: defect: RuntimeError: can't start new thread (due to memlimit) (closed: fixed)">#31395</a> removed <code>memlimit</code> for doctests.
</p>
</blockquote>
<p>
Should we close this then?
</p>
TicketMarkus WageringelFri, 09 Apr 2021 19:51:07 GMTstatus changed
https://trac.sagemath.org/ticket/28106#comment:46
https://trac.sagemath.org/ticket/28106#comment:46
<ul>
<li><strong>status</strong>
changed from <em>needs_work</em> to <em>needs_review</em>
</li>
</ul>
TicketDima PasechnikFri, 09 Apr 2021 20:21:43 GMTstatus, reviewer changed
https://trac.sagemath.org/ticket/28106#comment:47
https://trac.sagemath.org/ticket/28106#comment:47
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>positive_review</em>
</li>
<li><strong>reviewer</strong>
changed from <em>Volker Braun</em> to <em>Volker Braun, Dima Pasechnik</em>
</li>
</ul>
TicketMatthias KöppeThu, 24 Jun 2021 20:15:37 GMTstatus changed; resolution set
https://trac.sagemath.org/ticket/28106#comment:48
https://trac.sagemath.org/ticket/28106#comment:48
<ul>
<li><strong>status</strong>
changed from <em>positive_review</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>invalid</em>
</li>
</ul>
Ticket