Opened 9 years ago
Last modified 4 months ago
#8254 needs_work defect
Use lazy imports to speed up sage startup time
Reported by: | was | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | sage-8.5 |
Component: | misc | Keywords: | sd32 |
Cc: | Merged in: | ||
Authors: | Robert Bradshaw | Reviewers: | |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
Attachments (1)
Change History (26)
comment:1 Changed 9 years ago by
comment:2 Changed 9 years ago by
By "help," I mean "help to identify the problem(s)."
comment:3 Changed 9 years ago by
OS X for one.
comment:4 Changed 9 years ago by
FWIW, Sage 4.3.3 (with some mods to get it to build on Solaris) takes 8 seconds to start on a Sun Blade 1000, with 2 x 900 MHz CPUs and 2 GB RAM. That machine has 15,000 rpm 2 Gbit/s fibre channel FC-AL disks.
Considering the age of that machine, I'm not overly concerned over that one. Although this machine is quite old, the disks are quite high spec. I don't know if that gives any clues. File systems are local.
The doc test
sage -t "devel/sage/sage/rings/polynomial/symmetric_ideal.py"
takes 459.4 seconds to run on that machine, so this is no quick machine - I had to increase SAGE_TIMEOUT just to get some doctests to pass.
Dave
comment:5 Changed 9 years ago by
Given the startup time for me, on that SPARC, and the fact I have a quicker SPARC, I'm not overly concerned by this myself.
But those interested in fixing it might like to look at using 'iostat', 'vmstat' and 'sar' to see what the system is doing.
Dave
comment:6 Changed 9 years ago by
- Owner changed from tbd to drkirkby
PS, that Blade 1000 of mine is using UFS disks, not ZFS. Using ZFS would take more memory from a machine that already has very little.
I suspect the startup time has more to do with disk & file-system performance than it does with CPU speed.
Dave
comment:7 Changed 9 years ago by
- Owner changed from drkirkby to (none)
comment:8 Changed 8 years ago by
- Status changed from new to needs_review
I talked to Robert today and he told me thought he had marked this "needs review", but he hadn't. So I am.
comment:9 Changed 8 years ago by
Yep, this is ready for review (assuming it hasn't bitrotted). Regarding Davve Kirkby's comment, I agree that disk access is probably the main bottleneck, and this addresses that by loading fewer modules.
comment:10 Changed 8 years ago by
The patch applies to 4.6 after rebasing #8456
In my computer, from a first running (disc access is necessary), Sage passes from 12.059 seconds to 11.290 So, a save less than 10%
If I further lazy import doing a lazy import for every * imports and running sage.misc.lazy_import.save_cache_file() the time drops to 10.259
If I run Sage with the initialitation files in RAM (disc acces is not necessary) the time with lazy_importing all * drops from 1.123 to 0.907
So, still, with lazy imports, disc access is too high.
with Robert patch I get the following doctest failures: sage-devel -t -long -force_lib devel/sage/doc/en/bordeaux_2008/generators_for_rings.rst sage-devel -t -long -force_lib devel/sage/sage/structure/coerce_actions.pyx sage-devel -t -long -force_lib devel/sage/sage/modular/cusps.py sage-devel -t -long -force_lib devel/sage/sage/modular/hecke/module.py sage-devel -t -long -force_lib devel/sage/sage/modular/hecke/hecke_operator.py sage-devel -t -long -force_lib devel/sage/sage/modular/modsym/heilbronn.pyx sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/arithgroup_element.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gammaH.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_sl2z.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gamma0.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gamma1.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_generic.py sage-devel -t -long -force_lib devel/sage/sage/modular/modform/space.py sage-devel -t -long -force_lib devel/sage/sage/modular/modform/find_generators.py sage-devel -t -long -force_lib devel/sage/sage/modular/modform/cuspidal_submodule.py
comment:11 Changed 8 years ago by
- Status changed from needs_review to needs_work
The first import * (after every sage -b) needs to import the modules to get the names. However, if there's only a 10% gain, I should take another look and make sure something isn't accidentally still pulling it in. (A lot has changed since I wrote this patch...)
comment:12 Changed 8 years ago by
I've attached a patch that seems to get some of the worst offenders for wasteful imports in 4.6. I haven't run doctests yet, though...
comment:13 Changed 8 years ago by
From a warm file cache, my patch seems to shave off about 1.4% of the startup time (times averaged over 5 runs). From a cold file cache, my guess is that it would be more than that because the unnecessary imports I take care of touch a lot of files.
comment:14 Changed 8 years ago by
Another issue I have found with this patch. With the lazy_imports in all.py, the namespace used in the instances is not the global namespace.
(after several runnings of sage, so it uses the cached database of functions)
sage: sloane_sequence <sage.misc.lazy_import.LazyImport object at 0x3e4d950> sage: sloane_sequence._get_object() <function sloane_sequence at 0x51a7230> sage: sloane_sequence <sage.misc.lazy_import.LazyImport object at 0x3e4d950>
It does not introduce sloane_sequence function in the global namespace but in sloane_sequence._namespace that seems to not be the global one.
comment:15 Changed 8 years ago by
- Description modified (diff)
- Summary changed from sage takes way too long to startup to Use lazy imports to speed up sage startup time
Okay, I guess because of the ticket title, I attached my patch. But the ticket title is way too general. I've made the title more descriptive, and moved my much simpler patch to #10220
comment:16 Changed 8 years ago by
Concerning the input on the global namespace in sage.all.py one can write
import __builtin__ G = __builtin__.__dict__ del __builtin__
and insert everything in the dictionary G.
But his looks like a very ugly hack.
comment:17 Changed 8 years ago by
What I wrote above is nonsense. I am adding names to the builtin namespace. Instead it should be _ip.user_ns, the namespace of the IPython session.
If we only do lazy_imports from sage.all we will not get an usable enviroment that load fasts.
If we add lazy_imports in other all.py files we will get problems messing with namespaces.
Suppose the following:
in sage.all:[br] from sage.rings.all import *
in sage.rings.all:[br] from sage.rings.polynomial.all import *
If we lazy_import sage.rings.polynomial.all names in sage.rings.all then we have to choose a namespace.
If the namespace chosen is the top one. When inserting the real object, it will be inserted in the main namespace, so, in the module sage.rings.all we will always have the lazy_import object.
If the namespace chosen is sage.rings.all then we will always have the lazy_object in the main namespace...
This is too technical for me to offer a solution at module level. One idea is to add other files called lazy_all.py. these will be modules that import/lazy_import names in the IPython top namespace. They are not intended for the library code, only to have a more fine grain control on what are lazy_importing or not at the start.
Comments? It is worth to give it a try?
comment:18 Changed 8 years ago by
If we do a lazy_import of x in sage.foo, and then someone does "from sage.foo import x" in m, I don't see any way to resolve the original x in m. It may be worth registering a "top-level" namespace that substitutions would be made into.
Note, however, that lazy imports are for library code as well.
comment:19 Changed 8 years ago by
- Description modified (diff)
I'm making this ticket into a meta-ticket, link from here to various speedups.
comment:20 Changed 8 years ago by
- Keywords sd32 added
comment:21 Changed 6 years ago by
- Milestone changed from sage-5.11 to sage-5.12
comment:22 Changed 5 years ago by
- Milestone changed from sage-6.1 to sage-6.2
comment:23 Changed 5 years ago by
- Milestone changed from sage-6.2 to sage-6.3
comment:24 Changed 5 years ago by
- Milestone changed from sage-6.3 to sage-6.4
comment:25 Changed 4 months ago by
- Milestone changed from sage-6.4 to sage-8.5
Could you provide some additional data? Which platforms? I assume that
sage -startuptime
does not help, or does it?