Sage: Ticket #8254: Use lazy imports to speed up sage startup time
https://trac.sagemath.org/ticket/8254
<p>
Sage still takes too long. 20 seconds, 30 seconds, 5 seconds, on various places... and it is all so painful. This patch attempts to speed up the startup time using lazy imports.
</p>
<p>
Specific patches at
</p>
<ul><li><a class="needs_work ticket" href="https://trac.sagemath.org/ticket/11040" title="enhancement: Lazily import schemes to save startuptime (needs_work)">#11040</a> - sage.schemes
</li><li><a class="needs_work ticket" href="https://trac.sagemath.org/ticket/11043" title="defect: Lazily import plot. (needs_work)">#11043</a> - sage.plot
</li></ul>en-usSagehttps://trac.sagemath.org/chrome/site/logo_sagemath_trac.png
https://trac.sagemath.org/ticket/8254
Trac 1.1.6mpatelSun, 14 Feb 2010 02:06:00 GMT
https://trac.sagemath.org/ticket/8254#comment:1
https://trac.sagemath.org/ticket/8254#comment:1
<p>
Could you provide some additional data? Which platforms? I assume that <code>sage -startuptime</code> does not help, or does it?
</p>
TicketmpatelSun, 14 Feb 2010 03:52:23 GMT
https://trac.sagemath.org/ticket/8254#comment:2
https://trac.sagemath.org/ticket/8254#comment:2
<p>
By "help," I mean "help to identify the problem(s)."
</p>
TicketrobertwbMon, 15 Feb 2010 19:39:17 GMT
https://trac.sagemath.org/ticket/8254#comment:3
https://trac.sagemath.org/ticket/8254#comment:3
<p>
OS X for one.
</p>
TicketdrkirkbySun, 28 Feb 2010 17:59:23 GMT
https://trac.sagemath.org/ticket/8254#comment:4
https://trac.sagemath.org/ticket/8254#comment:4
<p>
FWIW, Sage 4.3.3 (with some mods to get it to build on Solaris) takes 8 seconds to start on a Sun Blade 1000, with 2 x 900 MHz CPUs and 2 GB RAM. That machine has 15,000 rpm 2 Gbit/s fibre channel FC-AL disks.
</p>
<p>
Considering the age of that machine, I'm not overly concerned over that one. Although this machine is quite old, the disks are quite high spec. I don't know if that gives any clues. File systems are local.
</p>
<p>
The doc test
</p>
<pre class="wiki">sage -t "devel/sage/sage/rings/polynomial/symmetric_ideal.py"
</pre><p>
takes 459.4 seconds to run on that machine, so this is no quick machine - I had to increase SAGE_TIMEOUT just to get some doctests to pass.
</p>
<p>
Dave
</p>
TicketdrkirkbySun, 28 Feb 2010 18:04:24 GMT
https://trac.sagemath.org/ticket/8254#comment:5
https://trac.sagemath.org/ticket/8254#comment:5
<p>
Given the startup time for me, on that SPARC, and the fact I have a quicker SPARC, I'm not overly concerned by this myself.
</p>
<p>
But those interested in fixing it might like to look at using 'iostat', 'vmstat' and 'sar' to see what the system is doing.
</p>
<p>
Dave
</p>
TicketdrkirkbySun, 28 Feb 2010 18:15:23 GMTowner changed
https://trac.sagemath.org/ticket/8254#comment:6
https://trac.sagemath.org/ticket/8254#comment:6
<ul>
<li><strong>owner</strong>
changed from <em>tbd</em> to <em>drkirkby</em>
</li>
</ul>
<p>
PS, that Blade 1000 of mine is using UFS disks, not ZFS. Using ZFS would take more memory from a machine that already has very little.
</p>
<p>
I suspect the startup time has more to do with disk & file-system performance than it does with CPU speed.
</p>
<p>
Dave
</p>
TicketdrkirkbySun, 28 Feb 2010 18:15:41 GMTowner deleted
https://trac.sagemath.org/ticket/8254#comment:7
https://trac.sagemath.org/ticket/8254#comment:7
<ul>
<li><strong>owner</strong>
changed from <em>drkirkby</em> to <em>(none)</em>
</li>
</ul>
TicketrobertwbSat, 06 Mar 2010 09:46:47 GMTattachment set
https://trac.sagemath.org/ticket/8254
https://trac.sagemath.org/ticket/8254
<ul>
<li><strong>attachment</strong>
set to <em>8254-lazy-schemes-import.patch</em>
</li>
</ul>
<p>
Shaves off about 0.3 seconds for me. Depends on <a class="closed ticket" href="https://trac.sagemath.org/ticket/8456" title="enhancement: lazy import improvements (closed: fixed)">#8456</a>
</p>
TicketwasWed, 03 Nov 2010 04:38:43 GMTstatus changed
https://trac.sagemath.org/ticket/8254#comment:8
https://trac.sagemath.org/ticket/8254#comment:8
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>needs_review</em>
</li>
</ul>
<p>
I talked to Robert today and he told me thought he had marked this "needs review", but he hadn't. So I am.
</p>
TicketrobertwbWed, 03 Nov 2010 08:00:56 GMT
https://trac.sagemath.org/ticket/8254#comment:9
https://trac.sagemath.org/ticket/8254#comment:9
<p>
Yep, this is ready for review (assuming it hasn't bitrotted). Regarding Davve Kirkby's comment, I agree that disk access is probably the main bottleneck, and this addresses that by loading fewer modules.
</p>
TicketlftaberaThu, 04 Nov 2010 13:25:25 GMT
https://trac.sagemath.org/ticket/8254#comment:10
https://trac.sagemath.org/ticket/8254#comment:10
<p>
The patch applies to 4.6 after rebasing <a class="closed ticket" href="https://trac.sagemath.org/ticket/8456" title="enhancement: lazy import improvements (closed: fixed)">#8456</a>
</p>
<p>
In my computer, from a first running (disc access is necessary), Sage passes from 12.059 seconds to 11.290 So, a save less than 10%
</p>
<p>
If I further lazy import doing a
lazy import for every * imports and running sage.misc.lazy_import.save_cache_file() the time drops to 10.259
</p>
<p>
If I run Sage with the initialitation files in RAM (disc acces is not necessary) the time with lazy_importing all * drops from 1.123 to 0.907
</p>
<p>
So, still, with lazy imports, disc access is too high.
</p>
<p>
with Robert patch I get the following doctest failures:
sage-devel -t -long -force_lib devel/sage/doc/en/bordeaux_2008/generators_for_rings.rst
sage-devel -t -long -force_lib devel/sage/sage/structure/coerce_actions.pyx
sage-devel -t -long -force_lib devel/sage/sage/modular/cusps.py
sage-devel -t -long -force_lib devel/sage/sage/modular/hecke/module.py
sage-devel -t -long -force_lib devel/sage/sage/modular/hecke/hecke_operator.py
sage-devel -t -long -force_lib devel/sage/sage/modular/modsym/heilbronn.pyx
sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/arithgroup_element.py
sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gammaH.py
sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_sl2z.py
sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gamma0.py
sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gamma1.py
sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_generic.py
sage-devel -t -long -force_lib devel/sage/sage/modular/modform/space.py
sage-devel -t -long -force_lib devel/sage/sage/modular/modform/find_generators.py
sage-devel -t -long -force_lib devel/sage/sage/modular/modform/cuspidal_submodule.py
</p>
TicketrobertwbThu, 04 Nov 2010 18:04:26 GMTstatus changed
https://trac.sagemath.org/ticket/8254#comment:11
https://trac.sagemath.org/ticket/8254#comment:11
<ul>
<li><strong>status</strong>
changed from <em>needs_review</em> to <em>needs_work</em>
</li>
</ul>
<p>
The first import * (after every sage -b) needs to import the modules to get the names. However, if there's only a 10% gain, I should take another look and make sure something isn't accidentally still pulling it in. (A lot has changed since I wrote this patch...)
</p>
TicketjasonFri, 05 Nov 2010 03:37:45 GMT
https://trac.sagemath.org/ticket/8254#comment:12
https://trac.sagemath.org/ticket/8254#comment:12
<p>
I've attached a patch that seems to get some of the worst offenders for wasteful imports in 4.6. I haven't run doctests yet, though...
</p>
TicketjasonFri, 05 Nov 2010 03:39:35 GMT
https://trac.sagemath.org/ticket/8254#comment:13
https://trac.sagemath.org/ticket/8254#comment:13
<p>
From a warm file cache, my patch seems to shave off about 1.4% of the startup time (times averaged over 5 runs). From a cold file cache, my guess is that it would be more than that because the unnecessary imports I take care of touch a lot of files.
</p>
TicketlftaberaFri, 05 Nov 2010 08:07:40 GMT
https://trac.sagemath.org/ticket/8254#comment:14
https://trac.sagemath.org/ticket/8254#comment:14
<p>
Another issue I have found with this patch. With the lazy_imports in all.py, the namespace used in the instances is not the global namespace.
</p>
<p>
(after several runnings of sage, so it uses the cached database of functions)
</p>
<pre class="wiki">sage: sloane_sequence
<sage.misc.lazy_import.LazyImport object at 0x3e4d950>
sage: sloane_sequence._get_object()
<function sloane_sequence at 0x51a7230>
sage: sloane_sequence
<sage.misc.lazy_import.LazyImport object at 0x3e4d950>
</pre><p>
It does not introduce sloane_sequence function in the global namespace but in sloane_sequence._namespace that seems to not be the global one.
</p>
TicketjasonFri, 05 Nov 2010 12:36:24 GMTdescription, summary changed; author set
https://trac.sagemath.org/ticket/8254#comment:15
https://trac.sagemath.org/ticket/8254#comment:15
<ul>
<li><strong>author</strong>
set to <em>Robert Bradshaw</em>
</li>
<li><strong>description</strong>
modified (<a href="/ticket/8254?action=diff&version=15">diff</a>)
</li>
<li><strong>summary</strong>
changed from <em>sage takes way too long to startup</em> to <em>Use lazy imports to speed up sage startup time</em>
</li>
</ul>
<p>
Okay, I guess because of the ticket title, I attached my patch. But the ticket title is way too general. I've made the title more descriptive, and moved my much simpler patch to <a class="closed ticket" href="https://trac.sagemath.org/ticket/10220" title="enhancement: Unnecessary imports cause slower sage startup (closed: fixed)">#10220</a>
</p>
TicketlftaberaThu, 03 Mar 2011 10:02:47 GMT
https://trac.sagemath.org/ticket/8254#comment:16
https://trac.sagemath.org/ticket/8254#comment:16
<p>
Concerning the input on the global namespace in sage.all.py one can write
</p>
<pre class="wiki">import __builtin__
G = __builtin__.__dict__
del __builtin__
</pre><p>
and insert everything in the dictionary G.
</p>
<p>
But his looks like a very ugly hack.
</p>
TicketlftaberaThu, 03 Mar 2011 22:59:18 GMT
https://trac.sagemath.org/ticket/8254#comment:17
https://trac.sagemath.org/ticket/8254#comment:17
<p>
What I wrote above is nonsense. I am adding names to the <span class="underline">builtin</span> namespace. Instead it should be _ip.user_ns, the namespace of the IPython session.
</p>
<p>
If we only do lazy_imports from sage.all we will not get an usable enviroment that load fasts.
</p>
<p>
If we add lazy_imports in other all.py files we will get problems messing with namespaces.
</p>
<p>
Suppose the following:
</p>
<p>
in sage.all:[br]
from sage.rings.all import *
</p>
<p>
in sage.rings.all:[br]
from sage.rings.polynomial.all import *
</p>
<p>
If we lazy_import sage.rings.polynomial.all names in sage.rings.all then we have to choose a namespace.
</p>
<p>
If the namespace chosen is the top one. When inserting the real object, it will be inserted in the main namespace, so, in the module sage.rings.all we will always have the lazy_import object.
</p>
<p>
If the namespace chosen is sage.rings.all then we will always have the lazy_object in the main namespace...
</p>
<p>
This is too technical for me to offer a solution at module level. One idea is to add other files called lazy_all.py. these will be modules that import/lazy_import names in the IPython top namespace. They are not intended for the library code, only to have a more fine grain control on what are lazy_importing or not at the start.
</p>
<p>
Comments? It is worth to give it a try?
</p>
TicketrobertwbFri, 04 Mar 2011 05:49:20 GMT
https://trac.sagemath.org/ticket/8254#comment:18
https://trac.sagemath.org/ticket/8254#comment:18
<p>
If we do a lazy_import of x in sage.foo, and then someone does "from sage.foo import x" in m, I don't see any way to resolve the original x in m. It may be worth registering a "top-level" namespace that substitutions would be made into.
</p>
<p>
Note, however, that lazy imports are for library code as well.
</p>
TicketrobertwbSat, 26 Mar 2011 09:18:18 GMTdescription changed
https://trac.sagemath.org/ticket/8254#comment:19
https://trac.sagemath.org/ticket/8254#comment:19
<ul>
<li><strong>description</strong>
modified (<a href="/ticket/8254?action=diff&version=19">diff</a>)
</li>
</ul>
<p>
I'm making this ticket into a meta-ticket, link from here to various speedups.
</p>
TicketwasWed, 24 Aug 2011 23:29:58 GMTkeywords set
https://trac.sagemath.org/ticket/8254#comment:20
https://trac.sagemath.org/ticket/8254#comment:20
<ul>
<li><strong>keywords</strong>
<em>sd32</em> added
</li>
</ul>
TicketjdemeyerTue, 13 Aug 2013 15:35:53 GMTmilestone changed
https://trac.sagemath.org/ticket/8254#comment:21
https://trac.sagemath.org/ticket/8254#comment:21
<ul>
<li><strong>milestone</strong>
changed from <em>sage-5.11</em> to <em>sage-5.12</em>
</li>
</ul>
Ticketvbraun_spamThu, 30 Jan 2014 21:20:52 GMTmilestone changed
https://trac.sagemath.org/ticket/8254#comment:22
https://trac.sagemath.org/ticket/8254#comment:22
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.1</em> to <em>sage-6.2</em>
</li>
</ul>
Ticketvbraun_spamTue, 06 May 2014 15:20:58 GMTmilestone changed
https://trac.sagemath.org/ticket/8254#comment:23
https://trac.sagemath.org/ticket/8254#comment:23
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.2</em> to <em>sage-6.3</em>
</li>
</ul>
Ticketvbraun_spamSun, 10 Aug 2014 16:51:03 GMTmilestone changed
https://trac.sagemath.org/ticket/8254#comment:24
https://trac.sagemath.org/ticket/8254#comment:24
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.3</em> to <em>sage-6.4</em>
</li>
</ul>
TicketchapotonMon, 03 Dec 2018 17:07:09 GMTmilestone changed
https://trac.sagemath.org/ticket/8254#comment:25
https://trac.sagemath.org/ticket/8254#comment:25
<ul>
<li><strong>milestone</strong>
changed from <em>sage-6.4</em> to <em>sage-8.5</em>
</li>
</ul>
Ticket