doctest memlimit being hit in sage.combinat.tableau tests

[embray]

This is a follow-up to #27681 which is now closed, as it fixed one issue related to libgap and pexpect gap possibly interacting in bad ways.

However, the original issue, which for a while I could not reproduce, seems to be coming up again. For example if I run just:

$ ./sage -t --long src/sage/combinat/tableau.py

I get

too many failed tests, not using stored timings
Running doctests with ID 2019-07-03-13-48-33-78ca2800.
Git branch: u/embray/docker-python3
Using --optional=build,dochtml,memlimit,mpir,notedown,pandoc_attributes,python2,rst2ipynb,sage
Doctesting 1 file.
sage -t --long src/sage/combinat/tableau.py
**********************************************************************
File "src/sage/combinat/tableau.py", line 7751, in sage.combinat.tableau.StandardTableaux_size.cardinality
Failed example:
    StandardTableaux(50).cardinality()  # long time
Exception raised:
    Traceback (most recent call last):
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1105, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.tableau.StandardTableaux_size.cardinality[4]>", line 1, in <module>
        StandardTableaux(Integer(50)).cardinality()  # long time
      File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3691)
        return self.get_object()(*args, **kwds)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
        return cls.classcall(cls, *args, **kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7584, in __classcall_private__
        return StandardTableaux_size(n)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
        return cls.classcall(cls, *args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
        w = self.f(*args, **kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
        instance = typecall(cls, *args, **options)
      File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
        return (<PyTypeObject*>type).tp_call(cls, args, kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7693, in __init__
        facade=True, keepkey=False)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/disjoint_union_enumerated_sets.py", line 288, in __init__
        self._facade_for = tuple(family)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1062, in __iter__
        yield self[i]
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1078, in __getitem__
        return self.function(i)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
        return cls.classcall(cls, *args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
        w = self.f(*args, **kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
        instance = typecall(cls, *args, **options)
      File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
        return (<PyTypeObject*>type).tp_call(cls, args, kwds)
    MemoryError
**********************************************************************
File "src/sage/combinat/tableau.py", line 8572, in sage.combinat.tableau.IncreasingTableaux.__init__
Failed example:
    TestSuite(S).run()  # long time
Exception raised:
    Traceback (most recent call last):
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1118, in compile_and_execute
        self.update_digests(example)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1009, in update_digests
        digest.update(str_to_bytes(reduce_hex(gen), 'ascii'))
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/parsing.py", line 442, in reduce_hex
        from operator import xor
    MemoryError
Process DocTestWorker-1:
Traceback (most recent call last):
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2149, in run
    task(self.options, self.outtmpfile, msgpipe, self.result_queue)
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2508, in __call__
    result_queue.put(result, False)
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/multiprocessing/queues.py", line 107, in put
    self._start_thread()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/multiprocessing/queues.py", line 195, in _start_thread
    self._thread.start()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/threading.py", line 736, in start
    _start_new_thread(self.__bootstrap, ())
error: can't start new thread
    Bad exit: 1

There is something about sage.combinat.tableau that the problem is always occurring there, though I'm not sure why. It's usually crashing on the same tests, but not always with the exact same traceback, suggesting some pseudo-deterministic memory corruption.

[embray]

On #27681 it was reported that the error doesn't occur when running the tests with --serial, but my experience is different. I still get:

$ ./sage -t --serial --long src/sage/combinat/tableau.py
too many failed tests, not using stored timings
Running doctests with ID 2019-07-03-13-55-40-8a0bf5f9.
Git branch: u/embray/docker-python3
Using --optional=build,dochtml,memlimit,mpir,notedown,pandoc_attributes,python2,rst2ipynb,sage
Doctesting 1 file.
sage -t --long src/sage/combinat/tableau.py
**********************************************************************
File "src/sage/combinat/tableau.py", line 7751, in sage.combinat.tableau.StandardTableaux_size.cardinality
Failed example:
    StandardTableaux(50).cardinality()  # long time
Exception raised:
    Traceback (most recent call last):
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1105, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.tableau.StandardTableaux_size.cardinality[4]>", line 1, in <module>
        StandardTableaux(Integer(50)).cardinality()  # long time
      File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3691)
        return self.get_object()(*args, **kwds)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
        return cls.classcall(cls, *args, **kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7584, in __classcall_private__
        return StandardTableaux_size(n)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
        return cls.classcall(cls, *args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
        w = self.f(*args, **kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
        instance = typecall(cls, *args, **options)
      File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
        return (<PyTypeObject*>type).tp_call(cls, args, kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7693, in __init__
        facade=True, keepkey=False)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/disjoint_union_enumerated_sets.py", line 288, in __init__
        self._facade_for = tuple(family)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1062, in __iter__
        yield self[i]
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/sets/family.py", line 1078, in __getitem__
        return self.function(i)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1735)
        return cls.classcall(cls, *args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1003, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6011)
        w = self.f(*args, **kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
        instance = typecall(cls, *args, **options)
      File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2185)
        return (<PyTypeObject*>type).tp_call(cls, args, kwds)
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7869, in __init__
        super(StandardTableaux_shape, self).__init__(category=FiniteEnumeratedSets())
      File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 6052, in __init__
        self.max_entry = None
    MemoryError
----------------------------------------------------------------------
Doctests interrupted: 0/1 files tested
----------------------------------------------------------------------
Total time for all tests: 59.6 seconds
    cpu time: 0.0 seconds
    cumulative wall time: 0.0 seconds
Traceback (most recent call last):
  File "/home/embray/src/sagemath/sage/src/bin/sage-runtests", line 179, in <module>
    err = DC.run()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 1232, in run
    self.run_doctests()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/control.py", line 933, in run_doctests
    self.dispatcher.dispatch()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 2008, in dispatch
    self.serial_dispatch()
  File "/home/embray/src/sagemath/sage/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1671, in serial_dispatch
    output = bytes_to_str(outtmpfile.read())
MemoryError

Since this is a pretty random issue, it possible that running with --serial just happened to make it go away in their case.

[embray]

At this point I'm not even certain it has anything to with libgap. That's just the context in which this first started appearing for me. It could also be a CPython bug.

[embray]

Sometimes I forget that on Linux we run the tests by default with --memlimit=3300 which sets RLIMIT_AS.

I wonder if that's all it is.

[embray]

Yep, the crash is just occurring upon hitting the 3300MB memlimit. Simple and mundane as that.

It could still be indirectly related to libgap. When investigating this earlier in #27681 we thought it might have something to do with GAP workspaces. This is because I first discovered the problem while working on #18267, in which libgap is used much more extensively directly in sage (as opposed to pexpect gap which is running outside the main sage process and so doesn't hit this memory limit as easily). It also seemed to go away if I deleted my existing GAP workspaces.

It could be that in the previous case, merely loading a pre-existing GAP workspace was enough to push over the memory limit.

[embray]

Of note: With the default of --memlimit=3300, roughly 2GB of virtual memory are already reserved by various things just while importing Sage. Of that, about 250MB is used just by Python and various loaded modules. 825 MB (i.e. virtual_memory_limit() // 4) are reserved by Pari, which is loaded no matter what. The rest I can't easily account for yet...

While running the tests, libgap reserves initially what amounts to virtual_memory_limit() // 10, or only 330 MB, but the 2 GB I mentioned previously doesn't even include libgap.

I wonder, if nothing else, as long as we're capping the virtual memory limit for doctests, the default amount reserved for Pari should be lower.

[embray]

One little thing we can do that happens to be good-enough for this case, it seems, is to further limit the amount of workspace Pari allocates by default, just when running the test suite.

Since many tests won't require that much memory for Pari this seems sensible.

In the case of this particular test the other issue is just reducing the amount of memory used when instantiating large Tableaux classes. I have opened #28114 for that.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​52b922f

Ticket #28106: Limit the initial memory allocation for Pari even further when running tests.

[tscrim]

I am a little split on this. I agree that we don't really need it for the doctests, but I don't like making the behavior different than what is done within a Sage session. Or is there something special with Cygwin and/or the doctest framework going on here?

[gh-mwageringel]

This problem can also be observed with a doctest for divisors() in sage/rings/integer.pyx. See #28162.

I tried the branch here, but it does not solve the issue for me. Both the tests in tableau.py and integer.pyx still fail.

On the other hand, raising the memlimit from 3300 to 3400 makes the tests pass for both of the files, without this branch even. If I understand the change correctly, it should decrease memory usage by much more than 100 MB, so I am not sure what is going on.

[vbraun]

I'm also hitting this on the buildbot, unsurprisingly.

IMHO its reasonable to provide less memory specifically for doctests, if only to encourage authors to limit resource usage when writing tests.

[vbraun]

I'm still running sometimes out of memory

Using --optional=4ti2,benzene,buckygen,build,cmake,database_cremona_ellcurve,database_jones_numfield,database_kohel,database_mutation_class,database_odlyzko_zeta,database_stein_watkins,database_stein_watkins_mini,database_symbolic_data,deformation,dochtml,e_antic,gambit,latte_int,libogg,lidia,lrslib,memlimit,mpir,normaliz,ore_algebra,pari_elldata,pari_galpol,pari_nftables,pari_seadata,plantri,python2,qhull,saclib,sage,tides,topcom
Doctesting entire Sage library.
Sorting sources by runtime so that slower doctests are run first....
Doctesting 3816 files using 16 threads.
**********************************************************************
File "src/sage/combinat/tableau.py", line 7698, in sage.combinat.tableau.StandardTableaux_size.cardinality
Failed example:
    StandardTableaux(50).cardinality()  # long time
Exception raised:
    Traceback (most recent call last):
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 681, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 1105, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.combinat.tableau.StandardTableaux_size.cardinality[4]>", line 1, in <module>
        StandardTableaux(Integer(50)).cardinality()  # long time
      File "sage/misc/lazy_import.pyx", line 354, in sage.misc.lazy_import.LazyImport.__call__ (build/cythonized/sage/misc/lazy_import.c:3697)
        return self.get_object()(*args, **kwds)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1741)
        return cls.classcall(cls, *args, **kwds)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7531, in __classcall_private__
        return StandardTableaux_size(n)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1741)
        return cls.classcall(cls, *args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1002, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6017)
        w = self.f(*args, **kwds)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
        instance = typecall(cls, *args, **options)
      File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2191)
        return (<PyTypeObject*>type).tp_call(cls, args, kwds)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7640, in __init__
        facade=True, keepkey=False)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/sets/disjoint_union_enumerated_sets.py", line 288, in __init__
        self._facade_for = tuple(family)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/sets/family.py", line 1062, in __iter__
        yield self[i]
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/sets/family.py", line 1078, in __getitem__
        return self.function(i)
      File "sage/misc/classcall_metaclass.pyx", line 335, in sage.misc.classcall_metaclass.ClasscallMetaclass.__call__ (build/cythonized/sage/misc/classcall_metaclass.c:1741)
        return cls.classcall(cls, *args, **kwds)
      File "sage/misc/cachefunc.pyx", line 1002, in sage.misc.cachefunc.CachedFunction.__call__ (build/cythonized/sage/misc/cachefunc.c:6017)
        w = self.f(*args, **kwds)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/structure/unique_representation.py", line 1027, in __classcall__
        instance = typecall(cls, *args, **options)
      File "sage/misc/classcall_metaclass.pyx", line 506, in sage.misc.classcall_metaclass.typecall (build/cythonized/sage/misc/classcall_metaclass.c:2191)
        return (<PyTypeObject*>type).tp_call(cls, args, kwds)
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 7816, in __init__
        super(StandardTableaux_shape, self).__init__(category=FiniteEnumeratedSets())
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/combinat/tableau.py", line 6000, in __init__
        Tableaux.__init__(self, **kwds)
      File "sage/structure/parent.pyx", line 296, in sage.structure.parent.Parent.__init__ (build/cythonized/sage/structure/parent.c:4589)
        CategoryObject.__init__(self, category, base)
      File "sage/structure/category_object.pyx", line 115, in sage.structure.category_object.CategoryObject.__init__ (build/cythonized/sage/structure/category_object.c:2053)
        self._init_category_(category)
      File "sage/structure/parent.pyx", line 339, in sage.structure.parent.Parent._init_category_ (build/cythonized/sage/structure/parent.c:5059)
        self.__class__ = dynamic_class(
      File "/var/lib/buildbot/slave/sage_git/build/local/lib/python2.7/site-packages/sage/structure/dynamic_class.py", line 335, in dynamic_class
        return dynamic_class_internal(name, bases, cls, reduction, doccls, prepend_cls_bases)
    MemoryError
**********************************************************************

[dimpase]

I still see this on my Fedora desktop, as I run tests with --long on. There seems to be something Fedora-specific here.

[vbraun]

Can we just reduce StandardTableaux(50) to something smaller here?

This is on Debian fwiw.

[dimpase]

hmm, but it is fast (for me, on Debian 10):

sage: timeit("StandardTableaux(50).cardinality()")
5 loops, best of 3: 169 μs per loop

It is not constructing this huge set for sure...

[tscrim]

Replying to dimpase:

hmm, but it is fast (for me, on Debian 10):

sage: timeit("StandardTableaux(50).cardinality()")
5 loops, best of 3: 169 μs per loop

It is not constructing this huge set for sure...

Which is the point of this doctest, the set is huge and it is using an actual computation. However, StandardPartitions does need to compute the partitions of 50 as part of the __init__ as part of this line:

Family(Partitions_n(n), StandardTableaux_shape),

Now the Family is created lazily; however, this gets expanded by the call self._facade_for = tuple(family) in the DisjointUnionEnumeratedSets.__init__. So we construct a set of size

sage: Partitions(50).cardinality()
204226

instead of

sage: ST = StandardTableaux(50)
sage: ST.cardinality()
27886995605342342839104615869259776

Now unfortunately I do not see a simple way to keep that family done lazily. However, changing it to 40 should still do an appropriate test but uses far less memory:

sage: Partitions(40).cardinality()
37338
sage: StandardTableaux(40).cardinality()
72682301192087742711233536

[gh-mwageringel]

The current branch does not work as intended because DOCTEST_MODE is not set yet when Pari is initialized. Perhaps it could be set earlier on, in sage-runtests?

[gh-mwageringel]

Replying to tscrim:

Now unfortunately I do not see a simple way to keep that family done lazily.

The following seems to work. All tests pass for me.

src/sage/categories/facade_sets.py

@@ -177 +177 @@
             if parents is True:
                 return True
             from sage.structure.element import parent
-            return parent(element) in parents
+            # parents is a tuple or (finite lazy) family of Parent types;
+            # LazyFamily does not implement __contains__, so we iterate over it
+            return parent(element) in iter(parents)
 
         def __contains__(self, element):
             """

src/sage/sets/disjoint_union_enumerated_sets.py

@@ -285 +285 @@
         self._facade  = facade
         if facade:
             if family in FiniteEnumeratedSets():
-                self._facade_for = tuple(family)
+                self._facade_for = family
             else:
                 # This allows the test suite to pass its tests by essentially
                 #   stating that this is a facade for any parent. Technically
@@ -576 +576 @@
                 raise ValueError("cannot coerce `%s` in the parent `%s`"%(el[1], P))
 
         # Check first to see if the parent of el is in the family
-        if (isinstance(el, Element) and isinstance(self._facade_for, tuple)
-            and el.parent() in self._facade_for):
+        if (isinstance(el, Element) and self._facade_for is not True
+            and el.parent() in iter(self._facade_for)):
             return el
 
         for P in self._family:

This change does not have the potential to solve the memory problems in integer.pyx, but it could be useful regardless of the memory issues. With this we could even test for StandardTableaux(500).cardinality() instead, which only takes a few milliseconds.

[tscrim]

Hmm...I thought I had some reason for not doing that when I looked at it for comment:15, but the logic all seems to checkout. There is one small behavior change: If it was given a finite iterator before, it would know it is a finite enumerated set, but now that doesn't happen (infinite iterators would loop forever before)

sage: iter([1,2,3,4,5]) in FiniteEnumeratedSets()
False

So I think that is a safe change.

[gh-mwageringel]

Replying to tscrim:

There is one small behavior change: If it was given a finite iterator before, it would know it is a finite enumerated set, but now that doesn't happen (infinite iterators would loop forever before)

I am not sure what you mean by the change in behavior. Could you give an example? The variable family is always a Family here, not an iterator, which is asserted by the __classcall_private__ in DisjointUnionEnumeratedSets.

However on second thought, instead of calling iter, it would be clearer to implement __contains__ for lazy families, which could then choose an appropriate behavior depending on whether the family is finite or not. When given a different kind of family, __contains__ could potentially do something more efficient than iterating. Currently, all other subclasses of Family already implement it, except for LazyFamily. This has the effect that the default implementation from Parent is called for lazy families, producing a very uninformative error message.

I could prepare a branch for this. Should I create a new ticket for that or put it here?

[tscrim]

Replying to gh-mwageringel:

Replying to tscrim:

There is one small behavior change: If it was given a finite iterator before, it would know it is a finite enumerated set, but now that doesn't happen (infinite iterators would loop forever before)

I am not sure what you mean by the change in behavior. Could you give an example? The variable family is always a Family here, not an iterator, which is asserted by the __classcall_private__ in DisjointUnionEnumeratedSets.

I mean the same input will result in a different object being created (i.e., the behavior) with your branch if it was given an iterator/generator. You are not just changing things that affect only DisjointUnionEnumeratedSets. The family change in facade_sets can affect many other things. Granted, doing things this way is arguably safer, but I am just pointing out a difference here.

However on second thought, instead of calling iter, it would be clearer to implement __contains__ for lazy families, which could then choose an appropriate behavior depending on whether the family is finite or not. When given a different kind of family, __contains__ could potentially do something more efficient than iterating. Currently, all other subclasses of Family already implement it, except for LazyFamily. This has the effect that the default implementation from Parent is called for lazy families, producing a very uninformative error message.

I could prepare a branch for this. Should I create a new ticket for that or put it here?

Please do so. Be aware, a lazy family may not know if it finite or not (a priori, I haven't checked this is actually true in the code). So that part might end up being more of a can of worms, but I don't know until someone actually does it.

[gh-mwageringel]

I opened #28273 implementing this. I created a new ticket for this, as Erik's approach tried here is justified in its own right.

Replying to tscrim:

I mean the same input will result in a different object being created (i.e., the behavior) with your branch if it was given an iterator/generator. You are not just changing things that affect only DisjointUnionEnumeratedSets. The family change in facade_sets can affect many other things. Granted, doing things this way is arguably safer, but I am just pointing out a difference here.

Ok, I see. Indeed the change in facade_sets could affect other things – though within the Sage library it should not affect anything, as _facade_for is always a tuple, currently. This assumption could easily break in the future though, so I think it is safer to do things as in #28273, which does not touch facade_sets.py at all.

[embray]

Replying to tscrim:

I am a little split on this. I agree that we don't really need it for the doctests, but I don't like making the behavior different than what is done within a Sage session. Or is there something special with Cygwin and/or the doctest framework going on here?

Yes, the fact that the doctest framework, by default, puts a low cap on virtual memory allocation allowed by the process, which would not normally be the case unless the user is manually setting those limits.

Therefore the large vmem allocation normally allowed for PARI's initial stack needs to be scaled back that much more when running the doctests in order to leave room for other things.

[embray]

Replying to gh-mwageringel:

The current branch does not work as intended because DOCTEST_MODE is not set yet when Pari is initialized. Perhaps it could be set earlier on, in sage-runtests?

I see, that is true. It's been several weeks since I last looked at this, but this change seemed to work for me, but now I can't be sure why. I must have been doing something differently at the time.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​65b276c	Ticket #28106: Limit the initial memory allocation for Pari even further when running tests.
​c6ff771	Fix accidental initialization of Pari during sage.doctest.parsing import

[embray]

I rebased the branch on current develop, and added a fix to ensure that pari was not being initialized so much earlier than intended during import of the doctest sub-package.

[gh-mwageringel]

Now there are several doctest failures as a result of the reduced Pari stack size, see the patchbot, so I am setting back to needs work. I have only had a brief look, but some of them seem hard to solve.

Also, considering that relying on a specific load order could be a bit fragile, would it be possible to add a doctest or an assertion to check that Pari is loaded after DOCTEST_MODE is set?

[embray]

Replying to gh-mwageringel:

Also, considering that relying on a specific load order could be a bit fragile, would it be possible to add a doctest or an assertion to check that Pari is loaded after DOCTEST_MODE is set?

I thought about that but I'm not sure exactly a good way to do that. It might be necessary to add some kind of pari_initialized flag.

[embray]

I wonder what the absolute minimum PARI stack size needed is. We shouldn't go below that, sure, but it's not really clear what the limit is here...

[dimpase]

Pari initialisation has 2 parameters, one is stack size, another some kind of “minimal prime”. I had to deal with it my recent branch on #28242...

[jdemeyer]

Replying to vbraun:

IMHO its reasonable to provide less memory specifically for doctests, if only to encourage authors to limit resource usage when writing tests.

I added that limit. My main motivation was to prevent accidents where Sage doctests would trash a machine by using all memory (that happened to me multiple times). The limit of 3300 MB was simply chosen as the smallest number that made all doctests pass at the time when that limit was imposed.

[jdemeyer]

Replying to embray:

Therefore the large vmem allocation normally allowed for PARI's initial stack needs to be scaled back that much more when running the doctests in order to leave room for other things.

I don't like adding yet another special case for doctests. Why not simply increase the 3300 MB memory limit a bit?

[jdemeyer]

Replying to embray:

Therefore the large vmem allocation normally allowed for PARI's initial stack needs to be scaled back that much more when running the doctests in order to leave room for other things.

I wouldn't be surprised if some tests (in particular heegner.py) used most of the available PARI stack. So I doubt that there is much room for making the PARI stack smaller during doctests.

[embray]

Replying to jdemeyer:

Replying to vbraun:

IMHO its reasonable to provide less memory specifically for doctests, if only to encourage authors to limit resource usage when writing tests.

I added that limit. My main motivation was to prevent accidents where Sage doctests would trash a machine by using all memory (that happened to me multiple times). The limit of 3300 MB was simply chosen as the smallest number that made all doctests pass at the time when that limit was imposed.

It just doesn't seem "fair", in a sense, that we impose a default hard limit on virtual memory allocation, and then have a large chunk of that eaten up by Pari, even when most of it isn't needed for most other tests.

When Pari runs out of memory isn't there some way to "catch" that, increase its stack, and re-try the operation? Perhaps tests/code that are expected to eat up the Pari stack should actually explicitly do that (it would be good to test that anyways).

[jdemeyer]

Replying to embray:

It just doesn't seem "fair", in a sense, that we impose a default hard limit on virtual memory allocation, and then have a large chunk of that eaten up by Pari, even when most of it isn't needed for most other tests.

I really don't understand at all what the word "fair" (even you put it in quotes) means here. This isn't about fairness, it's about solving a practical problem, namely preventing excessive memory usage by doctests. As I said: the limit of 3300 MB is chosen pretty arbitrarily and I wouldn't mind increasing it.

When Pari runs out of memory isn't there some way to "catch" that, increase its stack, and re-try the operation?

Sure, it probably could be done, but why bother? It's much easier to just allocate a large virtual stack. I just don't see the problem with that.

[vbraun]

Its also the only bound on actual ram usage during tests, and we should try to keep that manageable especially since tests tend to be run in parallel and cpu core count is increasing faster than ram prices are declining. Also, something <= 4gb is going to be the upper bound on 32-bit anyways.

[embray]

Replying to jdemeyer:

Replying to embray:

It just doesn't seem "fair", in a sense, that we impose a default hard limit on virtual memory allocation, and then have a large chunk of that eaten up by Pari, even when most of it isn't needed for most other tests.

I really don't understand at all what the word "fair" (even you put it in quotes) means here. This isn't about fairness, it's about solving a practical problem, namely preventing excessive memory usage by doctests. As I said: the limit of 3300 MB is chosen pretty arbitrarily and I wouldn't mind increasing it.

When Pari runs out of memory isn't there some way to "catch" that, increase its stack, and re-try the operation?

Sure, it probably could be done, but why bother? It's much easier to just allocate a large virtual stack. I just don't see the problem with that.

That's what I'm talking about: Normally it's fine to just allocate a large stack for PARI. But RLIMIT_AS limits the amount of virtual address space available to the process, and a large chunk of that gets used, without exception, for the PARI stack even for tests that don't need it so much, thus creating stricter limitations on what those tests are allowed to do. This is what I mean by "fair".

[embray]

I've since seen various other instances of this issue, not so much on my own machine, but others seem to be encountering it in various tests. Although I can't prove myself that this is the reason (they would have to demonstrate that the failing tests are hitting the VM limit, or that the problems go away with --memlimit=0.

But ISTM we need to do something about this. If not in the PARI interface I could also see about reducing the amount libgap needs to allocate, as I believe it's the increased usage of libgap in permutation groups that is largely responsible for the increase in issues like this.

[jdemeyer]

Why aren't we just doing the obvious thing and increasing the limit from 3300MB to 3400MB? There is nothing magical about that 3300MB limit, it's mostly an arbitrary number.

[embray]

Sure, I obviously wouldn't mind increasing the limit. I just wonder how sustainable that is. Perhaps a few more tests should be marked # optional - memlimit if this becomes a frequent problem.

It also makes me wonder if there aren't memory leaks that should be investigated.

[embray]

Ticket retargeted after milestone closed