Opened 13 years ago
Closed 11 years ago
#4631 closed defect (worksforme)
possible memory leak in matrix code?
Reported by: | craigcitro | Owned by: | mabshoff |
---|---|---|---|
Priority: | major | Milestone: | sage-4.3.1 |
Component: | memleak | Keywords: | |
Cc: | robertwb | Merged in: | |
Authors: | Reviewers: | ||
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
So I was looking at another ticket, and noticed the following memory usage of Hermite normal form:
sage: M = random_matrix(ZZ,50,50) sage: get_memory_usage() '1104M+' sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage() '1170M+'
It could be that the memory is getting "lost" in some other way, but it might be nice to see why it's happening. In fact, it's probably got nothing to do with HNF:
sage: get_memory_usage() '1170M+' sage: for _ in range(100): foo = M.echelon_form() ....: sage: get_memory_usage() '1237M+'
I'm happy to find out that there's a reasonable explanation for this.
Change History (8)
comment:1 Changed 13 years ago by
comment:2 Changed 13 years ago by
This may be relevant here. Either way it warrants consideration to call %reset at the end of Sage, but that would be another ticket.
On Mon, Dec 8, 2008 at 19:15, frank wang <f.yw@hotmail.com> wrote: > > Hi, > > > > I have a program with some variables consume a lot of memory. The first time > > I run it, it is fine. The second time I run it, I will get MemoryError. If I > > close the ipython and reopen it again, then I can run the program once. I am > > looking for a command to delete the intermediate variable once it is not > > used to save memory like in matlab clear command. How are you running this program? Be aware that IPython may be holding on to objects and preventing them from being deallocated. For example: In [7]: !cat memtest.py class A(object): def __del__(self): print 'Deleting %r' % self a = A() In [8]: %run memtest.py In [9]: %run memtest.py In [10]: %run memtest.py In [11]: del a In [12]: Do you really want to exit ([y]/n)? $ python memtest.py Deleting <__main__.A object at 0x915ab0> You can remove some of these references with %reset and maybe a gc.collect() for good measure. In [1]: %run memtest In [2]: %run memtest In [3]: %run memtest In [4]: %reset Once deleted, variables cannot be recovered. Proceed (y/[n])? y Deleting <__main__.A object at 0xf3e950> Deleting <__main__.A object at 0xf3e6d0> Deleting <__main__.A object at 0xf3e930> -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
comment:3 Changed 13 years ago by
- Cc robertwb added
Note that the latest upstream Cython 0.10.3 (merged in Sage 3.2.2.rc0 via #4798) improves things considerably:
---------------------------------------------------------------------- | Sage Version 3.2.2.alpha2, Release Date: 2008-12-13 | | Type notebook() for the GUI, and license() for information. | ---------------------------------------------------------------------- sage: M = random_matrix(ZZ,50,50) sage: get_memory_usage() 416.19921875 sage: for _ in range(100): foo = M.hermite_form() ....: sage: del foo sage: get_memory_usage() 423.74609375 sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage() 424.73828125 sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage() 425.703125
comment:4 Changed 13 years ago by
But #4639 does not improve this further. Note that "base" is the initial memory consumption:
sage: base=get_memory_usage() sage: M = random_matrix(ZZ,50,50) sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage()-base 7.5625 sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage()-base 8.515625 sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage()-base 9.5
Cheers,
Michael
comment:5 Changed 12 years ago by
#4883 might be relevant here.
comment:6 Changed 12 years ago by
The new Sage 3.2.3+eMPIRe build picked up the following leak in tut.py:
==5027== 8 bytes in 1 blocks are definitely lost in loss record 32 of 19,079 ==5027== at 0x4C22FEB: malloc (vg_replace_malloc.c:207) ==5027== by 0x84B3A4B: __gmpz_init_set_si (in /home/mabshoff/build/eMPIRe/sage-3.2.3.final/local/lib/libgmp.so.3.4.1) ==5027== by 0x271A110A: __pyx_pf_4sage_6matrix_4misc_matrix_integer_sparse_rational_reconstruction (misc.c:129 17) ==5027== by 0x417E92: PyObject_Call (abstract.c:1861) ==5027== by 0x25632FFA: __pyx_pf_4sage_6matrix_21matrix_integer_sparse_21Matrix_integer_sparse_rational_recons truction (matrix_integer_sparse.c:8698)==5027== by 0x417E92: PyObject_Call (abstract.c:1861) ==5027== by 0x271B2171: __pyx_pf_4sage_6matrix_4misc_matrix_rational_echelon_form_multimodular (misc.c:15034) ==5027== by 0x417E92: PyObject_Call (abstract.c:1861) ==5027== by 0x4816E1: PyEval_CallObjectWithKeywords (ceval.c:3442) ==5027== by 0x25CAAAE3: __pyx_pf_4sage_6matrix_22matrix_rational_sparse_22Matrix_rational_sparse__echelon_form _multimodular (matrix_rational_sparse.c:12416) ==5027== by 0x417E92: PyObject_Call (abstract.c:1861) ==5027== by 0x4816E1: PyEval_CallObjectWithKeywords (ceval.c:3442) ==5027== by 0x25CABA89: __pyx_pf_4sage_6matrix_22matrix_rational_sparse_22Matrix_rational_sparse_echelon_form (matrix_rational_sparse.c:12116)
William says:
[6:49pm] mabs: Yeah, I need to start poking around. [6:52pm] wstein: That could only leak if there is an exception raised somewhere in the body of the function. [6:52pm] wstein: In misc.pyx [6:53pm] wstein: So an exception must be raised. [6:54pm] wstein: maybe line 235 of matrix/misc.pyx could raise an exception, [6:54pm] wstein: hence return from that function without clearing gmp vars [6:58pm] mabs: mmhh
comment:7 Changed 12 years ago by
- Status changed from new to assigned
comment:8 Changed 11 years ago by
- Report Upstream set to N/A
- Resolution set to worksforme
- Status changed from new to closed
This seems to be fixed in sage-4.3:
sage: M = random_matrix(ZZ, 50, 50) sage: get_memory_usage() 182.859375 sage: for _ in range(100): foo = M.echelon_form() ....: sage: get_memory_usage() 183.32421875 sage: for _ in range(100): foo = M.hermite_form() ....: sage: get_memory_usage() 183.32421875
Given that it was vague to start with, I'm closing this as worksforme. (I tested on my machine and sage.math, with the same results.)
This might be related to #4639.
Cheers,
Michael