Opened 9 years ago

Last modified 4 months ago

#8254 needs_work defect

Use lazy imports to speed up sage startup time

Reported by: was Owned by:
Priority: major Milestone: sage-8.5
Component: misc Keywords: sd32
Cc: Merged in:
Authors: Robert Bradshaw Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by robertwb)

Sage still takes too long. 20 seconds, 30 seconds, 5 seconds, on various places... and it is all so painful. This patch attempts to speed up the startup time using lazy imports.

Specific patches at

Attachments (1)

8254-lazy-schemes-import.patch (1.6 KB) - added by robertwb 9 years ago.
Shaves off about 0.3 seconds for me. Depends on #8456

Download all attachments as: .zip

Change History (26)

comment:1 Changed 9 years ago by mpatel

Could you provide some additional data? Which platforms? I assume that sage -startuptime does not help, or does it?

comment:2 Changed 9 years ago by mpatel

By "help," I mean "help to identify the problem(s)."

comment:3 Changed 9 years ago by robertwb

OS X for one.

comment:4 Changed 9 years ago by drkirkby

FWIW, Sage 4.3.3 (with some mods to get it to build on Solaris) takes 8 seconds to start on a Sun Blade 1000, with 2 x 900 MHz CPUs and 2 GB RAM. That machine has 15,000 rpm 2 Gbit/s fibre channel FC-AL disks.

Considering the age of that machine, I'm not overly concerned over that one. Although this machine is quite old, the disks are quite high spec. I don't know if that gives any clues. File systems are local.

The doc test

sage -t  "devel/sage/sage/rings/polynomial/symmetric_ideal.py" 

takes 459.4 seconds to run on that machine, so this is no quick machine - I had to increase SAGE_TIMEOUT just to get some doctests to pass.

Dave

comment:5 Changed 9 years ago by drkirkby

Given the startup time for me, on that SPARC, and the fact I have a quicker SPARC, I'm not overly concerned by this myself.

But those interested in fixing it might like to look at using 'iostat', 'vmstat' and 'sar' to see what the system is doing.

Dave

comment:6 Changed 9 years ago by drkirkby

  • Owner changed from tbd to drkirkby

PS, that Blade 1000 of mine is using UFS disks, not ZFS. Using ZFS would take more memory from a machine that already has very little.

I suspect the startup time has more to do with disk & file-system performance than it does with CPU speed.

Dave

comment:7 Changed 9 years ago by drkirkby

  • Owner changed from drkirkby to (none)

Changed 9 years ago by robertwb

Shaves off about 0.3 seconds for me. Depends on #8456

comment:8 Changed 8 years ago by was

  • Status changed from new to needs_review

I talked to Robert today and he told me thought he had marked this "needs review", but he hadn't. So I am.

comment:9 Changed 8 years ago by robertwb

Yep, this is ready for review (assuming it hasn't bitrotted). Regarding Davve Kirkby's comment, I agree that disk access is probably the main bottleneck, and this addresses that by loading fewer modules.

comment:10 Changed 8 years ago by lftabera

The patch applies to 4.6 after rebasing #8456

In my computer, from a first running (disc access is necessary), Sage passes from 12.059 seconds to 11.290 So, a save less than 10%

If I further lazy import doing a lazy import for every * imports and running sage.misc.lazy_import.save_cache_file() the time drops to 10.259

If I run Sage with the initialitation files in RAM (disc acces is not necessary) the time with lazy_importing all * drops from 1.123 to 0.907

So, still, with lazy imports, disc access is too high.

with Robert patch I get the following doctest failures: sage-devel -t -long -force_lib devel/sage/doc/en/bordeaux_2008/generators_for_rings.rst sage-devel -t -long -force_lib devel/sage/sage/structure/coerce_actions.pyx sage-devel -t -long -force_lib devel/sage/sage/modular/cusps.py sage-devel -t -long -force_lib devel/sage/sage/modular/hecke/module.py sage-devel -t -long -force_lib devel/sage/sage/modular/hecke/hecke_operator.py sage-devel -t -long -force_lib devel/sage/sage/modular/modsym/heilbronn.pyx sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/arithgroup_element.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gammaH.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_sl2z.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gamma0.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_gamma1.py sage-devel -t -long -force_lib devel/sage/sage/modular/arithgroup/congroup_generic.py sage-devel -t -long -force_lib devel/sage/sage/modular/modform/space.py sage-devel -t -long -force_lib devel/sage/sage/modular/modform/find_generators.py sage-devel -t -long -force_lib devel/sage/sage/modular/modform/cuspidal_submodule.py

comment:11 Changed 8 years ago by robertwb

  • Status changed from needs_review to needs_work

The first import * (after every sage -b) needs to import the modules to get the names. However, if there's only a 10% gain, I should take another look and make sure something isn't accidentally still pulling it in. (A lot has changed since I wrote this patch...)

comment:12 Changed 8 years ago by jason

I've attached a patch that seems to get some of the worst offenders for wasteful imports in 4.6. I haven't run doctests yet, though...

comment:13 Changed 8 years ago by jason

From a warm file cache, my patch seems to shave off about 1.4% of the startup time (times averaged over 5 runs). From a cold file cache, my guess is that it would be more than that because the unnecessary imports I take care of touch a lot of files.

comment:14 Changed 8 years ago by lftabera

Another issue I have found with this patch. With the lazy_imports in all.py, the namespace used in the instances is not the global namespace.

(after several runnings of sage, so it uses the cached database of functions)

sage: sloane_sequence
<sage.misc.lazy_import.LazyImport object at 0x3e4d950>
sage: sloane_sequence._get_object()
<function sloane_sequence at 0x51a7230>
sage: sloane_sequence
<sage.misc.lazy_import.LazyImport object at 0x3e4d950>

It does not introduce sloane_sequence function in the global namespace but in sloane_sequence._namespace that seems to not be the global one.

comment:15 Changed 8 years ago by jason

  • Authors set to Robert Bradshaw
  • Description modified (diff)
  • Summary changed from sage takes way too long to startup to Use lazy imports to speed up sage startup time

Okay, I guess because of the ticket title, I attached my patch. But the ticket title is way too general. I've made the title more descriptive, and moved my much simpler patch to #10220

comment:16 Changed 8 years ago by lftabera

Concerning the input on the global namespace in sage.all.py one can write

import __builtin__
G = __builtin__.__dict__
del __builtin__

and insert everything in the dictionary G.

But his looks like a very ugly hack.

comment:17 Changed 8 years ago by lftabera

What I wrote above is nonsense. I am adding names to the builtin namespace. Instead it should be _ip.user_ns, the namespace of the IPython session.

If we only do lazy_imports from sage.all we will not get an usable enviroment that load fasts.

If we add lazy_imports in other all.py files we will get problems messing with namespaces.

Suppose the following:

in sage.all:[br] from sage.rings.all import *

in sage.rings.all:[br] from sage.rings.polynomial.all import *

If we lazy_import sage.rings.polynomial.all names in sage.rings.all then we have to choose a namespace.

If the namespace chosen is the top one. When inserting the real object, it will be inserted in the main namespace, so, in the module sage.rings.all we will always have the lazy_import object.

If the namespace chosen is sage.rings.all then we will always have the lazy_object in the main namespace...

This is too technical for me to offer a solution at module level. One idea is to add other files called lazy_all.py. these will be modules that import/lazy_import names in the IPython top namespace. They are not intended for the library code, only to have a more fine grain control on what are lazy_importing or not at the start.

Comments? It is worth to give it a try?

comment:18 Changed 8 years ago by robertwb

If we do a lazy_import of x in sage.foo, and then someone does "from sage.foo import x" in m, I don't see any way to resolve the original x in m. It may be worth registering a "top-level" namespace that substitutions would be made into.

Note, however, that lazy imports are for library code as well.

comment:19 Changed 8 years ago by robertwb

  • Description modified (diff)

I'm making this ticket into a meta-ticket, link from here to various speedups.

comment:20 Changed 8 years ago by was

  • Keywords sd32 added

comment:21 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:22 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:23 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:24 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4

comment:25 Changed 4 months ago by chapoton

  • Milestone changed from sage-6.4 to sage-8.5
Note: See TracTickets for help on using tickets.