Add checking memory functions

[jdemeyer]

We should introduce check_ variants of the Sage memory functions which raise MemoryError upon failure.

I also propose to add a BSD-inspired ​sage_reallocarray() function.

Attached branch introduces the functions and uses them in a few places as proof of concept.

[jdemeyer]

New commits:

​f81f349

Add memory functions

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​cc5edd2

Add memory functions

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​6ba47f0

Merge remote-tracking branch 'origin/develop' into ticket/10257

[mmezzarobba]

Also conflicts with #12212

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​307e526	Revert raise MemoryError changes from #12212
​cd8407b	Merge branch 'ticket/12212_revert_raise' into ticket/10257

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fa0612d

Merge remote-tracking branch 'origin/develop' into ticket/10257

[vbraun]

On OSX:

sage -t --long src/sage/modules/vector_rational_dense.pyx
**********************************************************************
File "src/sage/modules/vector_rational_dense.pyx", line 101, in sage.modules.vector_rational_dense.Vector_rational_dense._init
Failed example:
    try:
        Vector_rational_dense(QQ^(2^56))
    except (MemoryError, OverflowError):
        print "allocation failed"
Expected:
    allocation failed
Got:
    python(32767,0x7fff7d747300) malloc: *** mach_vm_map(size=2305843009213693952) failed (error code=3)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    allocation failed
**********************************************************************

[jdemeyer]

Volker, did you enable specific debugging flags? It looks like malloc is just verbosely saying that the allocation failed, it that the default on OS X?

[vbraun]

Seems to be the default on OSX, I didn't enable any special debugging.

[vbraun]

See also ​http://bugs.python.org/issue5614

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​c561e2e	Fix check_allocarray() failure doctest on OS X
​6650ca7	Add csage dependencies in Makefile
​c37e933	Merge ticket/17794 into ticket/10257

[dcoudert]

Hello,

Why not using this ?

cdef inline void* check_allocarray(size_t nmemb, size_t size) except? NULL:
   """
   Allocate memory for ``nmemb`` elements of size ``size``.
   """
   return check_malloc( mul_overflowcheck(nmemb, size) )


cdef inline void* check_malloc(size_t n) except? NULL:
   """
   Allocate ``n`` bytes of memory.
   """
   if n == 0:
      return NULL
   if n == <size_t>(-1):
      raise MemoryError("cannot allocate %s bytes" % n)
   cdef void* ret = sage_malloc(n)
   if ret == NULL:
      raise MemoryError("failed to allocate %s bytes" % n)
   return ret

At least, check_malloc should have the test if n==<size_t>-1 (if not already in sage_malloc), and other methods as well.

David.

[jdemeyer]

Replying to dcoudert:

Hello,

Why not using this ?

cdef inline void* check_allocarray(size_t nmemb, size_t size) except? NULL:
   """
   Allocate memory for ``nmemb`` elements of size ``size``.
   """
   return check_malloc( mul_overflowcheck(nmemb, size) )


cdef inline void* check_malloc(size_t n) except? NULL:
   """
   Allocate ``n`` bytes of memory.
   """
   if n == 0:
      return NULL
   if n == <size_t>(-1):
      raise MemoryError("cannot allocate %s bytes" % n)
   cdef void* ret = sage_malloc(n)
   if ret == NULL:
      raise MemoryError("failed to allocate %s bytes" % n)
   return ret

One reason is that the error message would be less clear this way: any overflow would lead to the same error message. With my code, the error message mentions exactly which allocation failed.

At least, check_malloc should have the test if n==<size_t>-1 (if not already in sage_malloc)

Why? The argument to malloc() is a number of bytes. Allocating <size_t>(-1) bytes will very likely fail anyway.

[dcoudert]

Replying to jdemeyer:

At least, check_malloc should have the test if n==<size_t>-1 (if not already in sage_malloc)

Why? The argument to malloc() is a number of bytes. Allocating <size_t>(-1) bytes will very likely fail anyway.

So we could avoid this test in check_allocarray, right?

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​352ffb8

Remove check for -1

[dcoudert]

Hello,

After a long compilation, I did a doc build and a sage -t --long src/sage/ (very very long...).

All tests pass on my MBA with OSX 10.10.

David.