Opened 9 years ago

Closed 8 years ago

#10801 closed enhancement (fixed)

Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade

Reported by: was Owned by: tbd
Priority: major Milestone: sage-4.7.2
Component: packages: standard Keywords: sd32
Cc: jason, kcrisman Merged in: sage-4.7.2.alpha3
Authors: William Stein, Keshav Kini Reviewers: Benjamin Jones, Keshav Kini
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by leif)

This is a frequently requested feature. We should start with some little script that does this, and build on it. I tried all the following, and ran the test suite and it worked fine.

  • rm SAGE_ROOT/local/lib/*.a
  • rm SAGE_ROOT/devel/sage/build/lib.*
  • rm SAGE_ROOT/devel/sage/build/temp.*
  • strip SAGE_ROOT/local/bin/Singular-* SAGE_ROOT/local/bin/gfan # gfan is a huge win.
  • jsmath image fonts are in MoinMoin? and are *HUGE*. Just delete everything related to moinmoin...
  • strip SAGE_ROOT/local/lib/*.so
  • rm -rf SAGE_ROOT/local/python/site-package/MoinMoin
  • rm all files in SAGE_ROOT/devel/sage/sage/ that begin "* Generated by Cython" (I didn't do that)

The patches below actually provide a new make target, "micro_release", rather than adding a new option to sage.


Apply trac_10801-root_repo.patch to the Sage root repository.

Apply trac_10801-local_bin_repo.patch to the Sage scripts repository.

Attachments (2)

trac_10801-root_repo.patch (695 bytes) - added by was 8 years ago.
apply to the repo in $SAGE_ROOT
trac_10801-local_bin_repo.patch (4.4 KB) - added by was 8 years ago.
replacing the patch by one that doesn't delete .a files.

Download all attachments as: .zip

Change History (31)

comment:1 Changed 9 years ago by jason

  • Cc jason added

It seems like the LiveCD folks have also worked on stuff like this...

comment:2 Changed 9 years ago by kini

If we mount a sage source installation with mount option strictatime or relatime we can just touch all the files to have an access time in the deep past (with something like cd $SAGE_ROOT ; for x in $(find) ; do touch -ad "1981-02-03 04:05:06" ; done), then run the whole doctest suite, then use stat() to find files which were unused. This should give us a good idea of what is never used. Unfortunately this does not seem possible on sage.math - POSIX atime is not respected in my homedir, probably as it is mounted over NFS...

comment:3 Changed 9 years ago by was

  • Description modified (diff)

comment:4 Changed 9 years ago by pi

if you're using linux, take a look at inotify-tools which has nice and clean command line tools, as it will give you a more fine-grained and realtime sense of what files are being used as they are accessed/modified to track down what's being used by which calls/tests/parts of sage.

comment:5 Changed 9 years ago by jhpalmieri

We can probably remove any Mercurial repositories.

comment:6 Changed 9 years ago by kcrisman

  • Cc kcrisman added

comment:7 Changed 9 years ago by kini

Here's some data I collected about what files are used and what files aren't, in sage 4.7.alpha3. Surprisingly many aren't, but come to think of it there's going to be a lot of stuff that users would want which isn't used by any doctest. I guess one should think of this data as a lower bound...

comment:8 follow-up: Changed 8 years ago by mhampton

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

comment:9 Changed 8 years ago by kini

Not delete, but strip. This removes debug symbols.

comment:10 Changed 8 years ago by leif

Slightly related: sage -crap ... (local/bin/sage-crap)

Note that we already have a few *clean targets in the top-level Makefile; there could be more (like strip), and some of the others could perhaps get improved or extended.

I personally wouldn't use -strip for anything else than really stripping debug symbols, i.e., if it is e.g. intended to delete also static libraries (which may be needed if someone installs some package, not necessarily an spkg), I'd choose a different name for such an option (or make target).

comment:11 Changed 8 years ago by was

After a long discussion, Keshav and I came up with:

   make micro_release

since this is a lot like "make release" with other software....

comment:12 in reply to: ↑ 8 Changed 8 years ago by was

Replying to mhampton:

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

I didn't suggest deleting gfan, but *stripping* it.

Changed 8 years ago by was

apply to the repo in $SAGE_ROOT

comment:13 Changed 8 years ago by was

Here's how I'm testing this:

wstein@sage:/tmp/wstein/sage-4.7.2.alpha1-sage.math.washington.edu-x86_64-Linux$
$ du -sh .
2.4G    .
$ du -s .
2468404 .

$ ./sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch
applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch

$ cd local/bin/; ../../sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch
applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch

$ time make micro_release
real    0m38.227s
user    0m2.370s
sys     0m5.140s

$ du -sh .
1.5G    .
$ du -s .
1547480 .

$ ./sage
# see it works

$ make ptestlong
# observe what happens (I will report back in a follow comment about this).

comment:14 Changed 8 years ago by was

  • Status changed from new to needs_review

ptestlong passes. W00t.

comment:15 follow-up: Changed 8 years ago by kini

Some thoughts:

  • In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.
  • The *_with_condition() functions should not talk about extensions in their docstrings.
  • What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

Going a bit farther, considering the target users will never develop or upgrade,

  • Should we delete mercurial repos?
  • Should we delete SPKGs?

comment:16 in reply to: ↑ 15 Changed 8 years ago by was

Replying to kini:

Some thoughts:

  • In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.

LATER: This could be added, but can we do it later in a future ticket. It would be good if the first version of this patch is quite safe. Hey, it already saves *1 GB* as is!

  • The *_with_condition() functions should not talk about extensions in their docstrings.

OK: I'll fix the patch right now.

  • What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

NO: Pointless, and scary in that it would break functionality, which is probably a bad idea. Good point about the size. I must have been testing something with some optional fonts installed.

Going a bit farther, considering the target users will never develop or upgrade,

  • Should we delete mercurial repos?

FUTURE: Yes, but in a future version of this?

  • Should we delete SPKGs?

FUTURE: It's safer to put a small string in each, to avoid confusion. This happens automatically when you build a binary with "sage -bdist". Anyway, this is something that could be added later.

comment:17 Changed 8 years ago by benjaminfjones

  • Reviewers set to Benjamin Jones

Tested the patches on sage.math and fresh build of Sage-4.7.1:

bjones@sage:/scratch/bjones/sage-4.7.1$ pwd
/scratch/bjones/sage-4.7.1
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.5G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2554916	.
bjones@sage:/scratch/bjones/sage-4.7.1$ time make micro_release
...
real	0m12.064s
user	0m2.560s
sys	0m7.600s
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.0G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2033236	.

So it saved 0.5 GB which is not the 1.0 GB observed by William above. Did you remove stuff by hand in that session that the script doesn't touch? Or maybe because I built the install from source instead of using a binary distribution?

I'm running make ptestlong now, will report back if anything fails.

comment:18 Changed 8 years ago by benjaminfjones

After producing the micro release, make ptestlong had one failure which disappeared when I doctested by hand:

The following tests failed:

        sage -t  -long -force_lib devel/sage/sage/tests/startup.py # 1 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1468.7 seconds
make: *** [ptestlong] Error 128

Subsequently by hand:

bjones@sage:/scratch/bjones/sage-4.7.1$ sage -t  -long -force_lib devel/sage/sage/tests/startup.py
sage -t -long -force_lib "devel/sage/sage/tests/startup.py" 
         [12.1 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 12.1 seconds

comment:19 Changed 8 years ago by was

Ben:

  1. Regarding the size savings, I started with a binary... so we may see a difference.
  1. Regarding the startup time failure, this has nothing to do with my patch. It routinely fails on sage.math. It's a test that it isn't taking more than 2 seconds on sage.math. I put it in back when we got the startup time down to solidly be < 1 second on sage.math, since I didn't want people to screw that up. They did though.

comment:20 Changed 8 years ago by was

  • Keywords sd32 added

comment:21 Changed 8 years ago by kini

  • Reviewers changed from Benjamin Jones to Benjamin Jones, Keshav Kini
  • Status changed from needs_review to positive_review

William: OK, sounds good. I'll make a ticket. Sounds like we're good to go here, no? Positive review from me.

comment:22 Changed 8 years ago by benjaminfjones

OK, I checked on OSX 10.6.8 binary install and I get results simular to William's. Looks good here too. Positive review from me as well.

comment:23 Changed 8 years ago by kini

New ticket is #11743.

comment:24 Changed 8 years ago by was

Martin Albrecht pointed out that deleting .a files could make it so users can't compile certain Cython programs, maybe. I'm not sure if we should care though, given the goal of this ticket ('never develop or upgrade').

comment:25 Changed 8 years ago by kini

Hm. If that includes %cython cells, I don't think that counts as "developing".

comment:26 Changed 8 years ago by was

Yes, I'm concerned. I'll simply comment out the line that deletes the .a files, and repost the patch.

Changed 8 years ago by was

replacing the patch by one that doesn't delete .a files.

comment:27 Changed 8 years ago by kini

  • Authors set to William Stein, Keshav Kini

Fair enough. Sadly this reduces our space savings by more than 200 MB.

(I'm putting myself as an author on this ticket because I seem to be listed as one the patch...)

comment:28 Changed 8 years ago by leif

  • Description modified (diff)

comment:29 Changed 8 years ago by leif

  • Merged in set to sage-4.7.2.alpha3
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.