Opened 12 years ago

Closed 12 years ago

#10548 closed defect (fixed)

The coercion model is keeping references to tracebacks which causes memory leaks.

Reported by: Maarten Derickx Owned by: Robert Bradshaw
Priority: major Milestone: sage-4.7
Component: coercion Keywords:
Cc: David Loeffler Merged in: sage-4.7.alpha4
Authors: Robert Bradshaw Reviewers: Maarten Derickx
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Jeroen Demeyer)

When I was doing some computations on modular forms, I noticed that the ModularSymbols_clear_cache() was not doing what it claims to do.

sage: import gc  
sage: m=ModularSymbols(Gamma1(29),sign=1)
sage: m=[]
sage: ModularSymbols_clear_cache()
sage: gc.collect()
57
sage: [x for x in gc.get_objects() if isinstance(x,sage.modular.modsym.ambient.ModularSymbolsAmbient_wtk_g1)]
[Modular Symbols space of dimension 49 for Gamma_1(29) of weight 2 with sign 1 and over Rational Field]
sage: 

So even after garbage collection the modular symbols element is still in memory.

When testing please also make sure that the last stackframe from #10570 is also gone.

Apply 10548-coerce-traceback.2.patch and trac_10548-coerce-traceback-doctest.v2.patch

Attachments (5)

chain.png (36.9 KB) - added by Maarten Derickx 12 years ago.
10548-coerce-traceback.patch (3.6 KB) - added by Robert Bradshaw 12 years ago.
10548-coerce-traceback.2.patch (3.5 KB) - added by Maarten Derickx 12 years ago.
rebased against 4.7.alpha1
10548-coerce-traceback-doctest.patch (1.0 KB) - added by Maarten Derickx 12 years ago.
trac_10548-coerce-traceback-doctest.v2.patch (1.7 KB) - added by Keshav Kini 12 years ago.
new doctest

Download all attachments as: .zip

Change History (27)

comment:1 Changed 12 years ago by Maarten Derickx

Ps. you might want to use smaller numbers then 97 to test because ModularSymbols?(Gamma1(97),sign=1) takes a while. It behaves the same, but with smaller memory usage for smaller numbers.

comment:2 Changed 12 years ago by Maarten Derickx

According to heapy the aditional memory comes from;

dict of sage.modular.modsym.manin_symbols.ManinSymbol?

mderickx@mod:~$ sage
----------------------------------------------------------------------
| Sage Version 4.6, Release Date: 2010-10-30                         |
| Type notebook() for the GUI, and license() for information.        |
----------------------------------------------------------------------
sage: import gc
sage: from guppy import hpy; hp=hpy()
sage: M=ModularSymbols(Gamma1(97),sign=1)
sage: get_memory_usage()
848.1953125
sage: M=[]
sage: get_memory_usage()
848.1953125
sage: ModularSymbols_clear_cache()
sage: get_memory_usage()
848.1953125
sage: hp.heap()
Partition of a set of 445281 objects. Total size = 68518056 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0 169534  38 28541776  42  28541776  42 str
     1 125494  28 10674696  16  39216472  57 tuple
     2   1624   0  3964480   6  43180952  63 dict of module
     3  25817   6  3098040   5  46278992  68 types.CodeType
     4  24634   6  2956080   4  49235072  72 function
     5   2798   1  2949584   4  52184656  76 dict (no owner)
     6   2660   1  2727392   4  54912048  80 dict of type
     7   9408   2  2634240   4  57546288  84 dict of sage.modular.modsym.manin_symbols.ManinSymbol
     8   2660   1  2388040   3  59934328  87 type
     9   4293   1  1010536   1  60944864  89 list
<895 more rows. Type e.g. '_.more' to view.>
sage: 

comment:3 Changed 12 years ago by Maarten Derickx

Description: modified (diff)

comment:4 Changed 12 years ago by Maarten Derickx

It's not an error of the ModularSymbols_clear_cache() function

sage: ModularSymbols(Gamma1(29),sign=1,use_cache=False)
Modular Symbols space of dimension 49 for Gamma_1(29) of weight 2 with sign 1 and over Rational Field
sage: None
sage: gc.collect()
54
sage: [x for x in gc.get_objects() if isinstance(x,sage.modular.modsym.ambient.ModularSymbolsAmbient_wtk_g1)]
[Modular Symbols space of dimension 49 for Gamma_1(29) of weight 2 with sign 1 and over Rational Field]

comment:5 Changed 12 years ago by Maarten Derickx

See the attached chain.png picture created with objgraph. There is a reference chain to the homset module. And the homeset module will not get destroyed so we have to find out where this chain comes from.

comment:6 Changed 12 years ago by David Loeffler

Might this be an instance of #715? Modular symbols spaces are Parents, so various maps in and out of them are probably getting cached by the coercion model machinery.

Changed 12 years ago by Maarten Derickx

Attachment: chain.png added

comment:7 Changed 12 years ago by Maarten Derickx

Cc: David Loeffler added

Sorry I failed in adding the mentioned chain.png before. It shows a reference chain explaining why ManinSymbol?(37,35) is still in the memory. It seems that it has at least to do something with code in the homset module. I don't know enough details about the coercion and category framework and their interaction to say that it has something to do with 715. But the description of 715 really looks like it could be the cause.

If it is coercion, it might also be related to: #10570

comment:8 Changed 12 years ago by David Loeffler

Having looked at #10570, I don't think that's the reason. #715 seems a much more likely culprit.

comment:9 Changed 12 years ago by Maarten Derickx

Description: modified (diff)

comment:10 Changed 12 years ago by Maarten Derickx

Here is some code run on sage 4.6.1 showing that it is really the coercion framework.

sage: import gc  
sage: import objgraph
sage: import inspect
sage: m=ModularSymbols(Gamma1(29),use_cache=False)
sage: m=[]
sage: ModularSymbols_clear_cache()
sage: gc.collect()
54
sage: a=[x for x in gc.get_objects() if isinstance(x,sage.modular.modsym.ambient.ModularSymbolsAmbient_wtk_g1)]
sage: l=objgraph.find_backref_chain(a[0],inspect.ismodule,max_depth=15,extra_ignore=[id(a)])
sage: map(type,l)
[<type 'module'>, <type 'dict'>, <type 'sage.structure.coerce.CoercionModel_cache_maps'>, <type 'list'>, <type 'tuple'>, <type 'traceback'>, <type 'frame'>, <type 'frame'>, <type 'frame'>, <type 'frame'>, <class 'sage.modular.modsym.ambient.ModularSymbolsAmbient_wtk_g1_with_category'>]

comment:11 Changed 12 years ago by Maarten Derickx

btw it goes wrong in the same way on 4.7.alpha1

comment:12 Changed 12 years ago by Maarten Derickx

Component: modular formscoercion
Owner: changed from Craig Citro to Robert Bradshaw
Summary: ModularSymbols_clear_cache() not clearing everything?The coercion model is keeping references to tracebacks which causes memory leaks.

comment:13 Changed 12 years ago by Maarten Derickx

Description: modified (diff)

Changed 12 years ago by Robert Bradshaw

comment:14 Changed 12 years ago by Robert Bradshaw

Status: newneeds_review

The attached patch fixes the issue described in the title, but does not fix #715.

Changed 12 years ago by Maarten Derickx

rebased against 4.7.alpha1

comment:15 Changed 12 years ago by Maarten Derickx

Status: needs_reviewpositive_review

Looks good to me.

Changed 12 years ago by Maarten Derickx

comment:16 Changed 12 years ago by Maarten Derickx

Status: positive_reviewneeds_work

comment:17 Changed 12 years ago by Maarten Derickx

Status: needs_workneeds_review

Robert, (or someone else), can you review the doctest I added?

comment:18 Changed 12 years ago by Maarten Derickx

Authors: Robert Bradshaw
Description: modified (diff)
Reviewers: Maarten Derickx

comment:19 Changed 12 years ago by Robert Bradshaw

Status: needs_reviewpositive_review

The doctest looks good to me.

Changed 12 years ago by Keshav Kini

new doctest

comment:20 Changed 12 years ago by Keshav Kini

Description: modified (diff)

I folded in the doctest from the current patch at #10570, also by Maarten.

comment:21 Changed 12 years ago by Jeroen Demeyer

Description: modified (diff)

comment:22 Changed 12 years ago by Jeroen Demeyer

Merged in: sage-4.7.alpha4
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.