Opened 8 years ago

Last modified 5 years ago

#11729 new enhancement

Cache module import locations

Reported by: vbraun Owned by: jason
Priority: major Milestone: sage-6.4
Component: misc Keywords: sd32
Cc: was, robertwb, ohanar, SimonKing Merged in:
Authors: Volker Braun Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by vbraun)

This ticket implements a cached module importer for Python. It compiles a list of all available files during build (when sage -b is run), hooks into the Python import framework and finds modules by the predetermined file list.

Building the Sage library (sage -b) automatically creates the cache and the sitecustomize.py that is used to load the importer into the Python session. The cache is a standard Python dictionary in sage/misc/modules_cache.py.

[vbraun@volker-laptop-two hg]$ strace -e file -f sage -c quit |& grep ENOENT | wc -l
26327
[vbraun@volker-laptop-two hg]$ sage -b
...
[vbraun@volker-laptop-two hg]$ strace -e file -f sage -c quit |& grep ENOENT | wc -l
3785

If you encounter any problems while testing this patch, you have to

rm $SAGE_LOCAL/lib/python/site-packages/sitecustomize.py*

to not load the importer during Python startup.

Apply trac_11729_cached_importer.patch to the sage library and trac_11729_import_cache_during_build.patch to the sage_scripts ($AGE_LOCAL/bin) repository.

Attachments (2)

trac_11729_import_cache_during_build.patch (1.3 KB) - added by vbraun 8 years ago.
Updated patch
trac_11729_cached_importer.patch (18.9 KB) - added by vbraun 8 years ago.
Updated patch

Download all attachments as: .zip

Change History (16)

comment:1 Changed 8 years ago by vbraun

  • Description modified (diff)

comment:2 Changed 8 years ago by vbraun

  • Cc was added

comment:3 Changed 8 years ago by robertwb

  • Cc robertwb added

comment:4 Changed 8 years ago by was

  • Keywords sd32 added

comment:5 Changed 8 years ago by vbraun

  • Description modified (diff)

comment:6 Changed 8 years ago by was

NOTE: If you try to test this, watch out because of these lines:

	113	cat <<EOF > "$SAGE_LOCAL/lib/python/site-packages/sitecustomize.py" 
 	114	from sage.misc.importer import sage_importer 
 	115	sage_importer.register() 
 	116	EOF 

They can make it so your Sage install is totally broken. To fix, just remove the contents of sitecustomize.py. I spent 15 minutes confused by this.

comment:7 Changed 8 years ago by vbraun

You have to delete both $SAGE_LOCAL/lib/python/site-packages/sitecustomize.py and $SAGE_LOCAL/lib/python/site-packages/sitecustomize.pyc, though the latter might not exist. The final version of the patch will test that sage starts up successfully and otherwise delete the sitecustomize.py automatically.

Also, if you run into problems (presumably some ImportError), please post what is causing it.

Changed 8 years ago by vbraun

Updated patch

Changed 8 years ago by vbraun

Updated patch

comment:8 Changed 8 years ago by vbraun

  • Description modified (diff)

I've added another workaround for PIL.Image, now make ptest passes! I would say that the patch is now feature-complete, though it lacks documentation and doctests. Though writing doctests is tricky without breaking the module importing. Somebody should test it on OSX :-)

comment:9 Changed 7 years ago by ohanar

  • Cc ohanar added

comment:10 Changed 7 years ago by SimonKing

  • Cc SimonKing added

comment:11 Changed 6 years ago by jdemeyer

  • Milestone changed from sage-5.11 to sage-5.12

comment:12 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.1 to sage-6.2

comment:13 Changed 6 years ago by vbraun_spam

  • Milestone changed from sage-6.2 to sage-6.3

comment:14 Changed 5 years ago by vbraun_spam

  • Milestone changed from sage-6.3 to sage-6.4
Note: See TracTickets for help on using tickets.