Opened 12 years ago

Closed 11 years ago

#10801 closed enhancement (fixed)

Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade

Reported by: William Stein Owned by: tbd
Priority: major Milestone: sage-4.7.2
Component: packages: standard Keywords: sd32
Cc: Jason Grout, Karl-Dieter Crisman Merged in: sage-4.7.2.alpha3
Authors: William Stein, Keshav Kini Reviewers: Benjamin Jones, Keshav Kini
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Leif Leonhardy)

This is a frequently requested feature. We should start with some little script that does this, and build on it. I tried all the following, and ran the test suite and it worked fine.

  • rm SAGE_ROOT/local/lib/*.a
  • rm SAGE_ROOT/devel/sage/build/lib.*
  • rm SAGE_ROOT/devel/sage/build/temp.*
  • strip SAGE_ROOT/local/bin/Singular-* SAGE_ROOT/local/bin/gfan # gfan is a huge win.
  • jsmath image fonts are in MoinMoin? and are *HUGE*. Just delete everything related to moinmoin...
  • strip SAGE_ROOT/local/lib/*.so
  • rm -rf SAGE_ROOT/local/python/site-package/MoinMoin
  • rm all files in SAGE_ROOT/devel/sage/sage/ that begin "* Generated by Cython" (I didn't do that)

The patches below actually provide a new make target, "micro_release", rather than adding a new option to sage.


Apply trac_10801-root_repo.patch to the Sage root repository.

Apply trac_10801-local_bin_repo.patch to the Sage scripts repository.

Attachments (2)

trac_10801-root_repo.patch (695 bytes) - added by William Stein 11 years ago.
apply to the repo in $SAGE_ROOT
trac_10801-local_bin_repo.patch (4.4 KB) - added by William Stein 11 years ago.
replacing the patch by one that doesn't delete .a files.

Download all attachments as: .zip

Change History (31)

comment:1 Changed 12 years ago by Jason Grout

Cc: Jason Grout added

It seems like the LiveCD folks have also worked on stuff like this...

comment:2 Changed 12 years ago by Keshav Kini

If we mount a sage source installation with mount option strictatime or relatime we can just touch all the files to have an access time in the deep past (with something like cd $SAGE_ROOT ; for x in $(find) ; do touch -ad "1981-02-03 04:05:06" ; done), then run the whole doctest suite, then use stat() to find files which were unused. This should give us a good idea of what is never used. Unfortunately this does not seem possible on sage.math - POSIX atime is not respected in my homedir, probably as it is mounted over NFS...

comment:3 Changed 12 years ago by William Stein

Description: modified (diff)

comment:4 Changed 12 years ago by Paul Ivanov

if you're using linux, take a look at inotify-tools which has nice and clean command line tools, as it will give you a more fine-grained and realtime sense of what files are being used as they are accessed/modified to track down what's being used by which calls/tests/parts of sage.

comment:5 Changed 12 years ago by John Palmieri

We can probably remove any Mercurial repositories.

comment:6 Changed 12 years ago by Karl-Dieter Crisman

Cc: Karl-Dieter Crisman added

comment:7 Changed 12 years ago by Keshav Kini

Here's some data I collected about what files are used and what files aren't, in sage 4.7.alpha3. Surprisingly many aren't, but come to think of it there's going to be a lot of stuff that users would want which isn't used by any doctest. I guess one should think of this data as a lower bound...

comment:8 Changed 11 years ago by mhampton

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

comment:9 Changed 11 years ago by Keshav Kini

Not delete, but strip. This removes debug symbols.

comment:10 Changed 11 years ago by Leif Leonhardy

Slightly related: sage -crap ... (local/bin/sage-crap)

Note that we already have a few *clean targets in the top-level Makefile; there could be more (like strip), and some of the others could perhaps get improved or extended.

I personally wouldn't use -strip for anything else than really stripping debug symbols, i.e., if it is e.g. intended to delete also static libraries (which may be needed if someone installs some package, not necessarily an spkg), I'd choose a different name for such an option (or make target).

comment:11 Changed 11 years ago by William Stein

After a long discussion, Keshav and I came up with:

   make micro_release

since this is a lot like "make release" with other software....

comment:12 in reply to:  8 Changed 11 years ago by William Stein

Replying to mhampton:

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

I didn't suggest deleting gfan, but *stripping* it.

Changed 11 years ago by William Stein

Attachment: trac_10801-root_repo.patch added

apply to the repo in $SAGE_ROOT

comment:13 Changed 11 years ago by William Stein

Here's how I'm testing this:

wstein@sage:/tmp/wstein/sage-4.7.2.alpha1-sage.math.washington.edu-x86_64-Linux$
$ du -sh .
2.4G    .
$ du -s .
2468404 .

$ ./sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch
applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch

$ cd local/bin/; ../../sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch
applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch

$ time make micro_release
real    0m38.227s
user    0m2.370s
sys     0m5.140s

$ du -sh .
1.5G    .
$ du -s .
1547480 .

$ ./sage
# see it works

$ make ptestlong
# observe what happens (I will report back in a follow comment about this).

comment:14 Changed 11 years ago by William Stein

Status: newneeds_review

ptestlong passes. W00t.

comment:15 Changed 11 years ago by Keshav Kini

Some thoughts:

  • In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.
  • The *_with_condition() functions should not talk about extensions in their docstrings.
  • What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

Going a bit farther, considering the target users will never develop or upgrade,

  • Should we delete mercurial repos?
  • Should we delete SPKGs?

comment:16 in reply to:  15 Changed 11 years ago by William Stein

Replying to kini:

Some thoughts:

  • In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.

LATER: This could be added, but can we do it later in a future ticket. It would be good if the first version of this patch is quite safe. Hey, it already saves *1 GB* as is!

  • The *_with_condition() functions should not talk about extensions in their docstrings.

OK: I'll fix the patch right now.

  • What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

NO: Pointless, and scary in that it would break functionality, which is probably a bad idea. Good point about the size. I must have been testing something with some optional fonts installed.

Going a bit farther, considering the target users will never develop or upgrade,

  • Should we delete mercurial repos?

FUTURE: Yes, but in a future version of this?

  • Should we delete SPKGs?

FUTURE: It's safer to put a small string in each, to avoid confusion. This happens automatically when you build a binary with "sage -bdist". Anyway, this is something that could be added later.

comment:17 Changed 11 years ago by Benjamin Jones

Reviewers: Benjamin Jones

Tested the patches on sage.math and fresh build of Sage-4.7.1:

bjones@sage:/scratch/bjones/sage-4.7.1$ pwd
/scratch/bjones/sage-4.7.1
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.5G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2554916	.
bjones@sage:/scratch/bjones/sage-4.7.1$ time make micro_release
...
real	0m12.064s
user	0m2.560s
sys	0m7.600s
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.0G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2033236	.

So it saved 0.5 GB which is not the 1.0 GB observed by William above. Did you remove stuff by hand in that session that the script doesn't touch? Or maybe because I built the install from source instead of using a binary distribution?

I'm running make ptestlong now, will report back if anything fails.

comment:18 Changed 11 years ago by Benjamin Jones

After producing the micro release, make ptestlong had one failure which disappeared when I doctested by hand:

The following tests failed:

        sage -t  -long -force_lib devel/sage/sage/tests/startup.py # 1 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1468.7 seconds
make: *** [ptestlong] Error 128

Subsequently by hand:

bjones@sage:/scratch/bjones/sage-4.7.1$ sage -t  -long -force_lib devel/sage/sage/tests/startup.py
sage -t -long -force_lib "devel/sage/sage/tests/startup.py" 
         [12.1 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 12.1 seconds

comment:19 Changed 11 years ago by William Stein

Ben:

  1. Regarding the size savings, I started with a binary... so we may see a difference.
  1. Regarding the startup time failure, this has nothing to do with my patch. It routinely fails on sage.math. It's a test that it isn't taking more than 2 seconds on sage.math. I put it in back when we got the startup time down to solidly be < 1 second on sage.math, since I didn't want people to screw that up. They did though.

comment:20 Changed 11 years ago by William Stein

Keywords: sd32 added

comment:21 Changed 11 years ago by Keshav Kini

Reviewers: Benjamin JonesBenjamin Jones, Keshav Kini
Status: needs_reviewpositive_review

William: OK, sounds good. I'll make a ticket. Sounds like we're good to go here, no? Positive review from me.

comment:22 Changed 11 years ago by Benjamin Jones

OK, I checked on OSX 10.6.8 binary install and I get results simular to William's. Looks good here too. Positive review from me as well.

comment:23 Changed 11 years ago by Keshav Kini

New ticket is #11743.

comment:24 Changed 11 years ago by William Stein

Martin Albrecht pointed out that deleting .a files could make it so users can't compile certain Cython programs, maybe. I'm not sure if we should care though, given the goal of this ticket ('never develop or upgrade').

comment:25 Changed 11 years ago by Keshav Kini

Hm. If that includes %cython cells, I don't think that counts as "developing".

comment:26 Changed 11 years ago by William Stein

Yes, I'm concerned. I'll simply comment out the line that deletes the .a files, and repost the patch.

Changed 11 years ago by William Stein

replacing the patch by one that doesn't delete .a files.

comment:27 Changed 11 years ago by Keshav Kini

Authors: William Stein, Keshav Kini

Fair enough. Sadly this reduces our space savings by more than 200 MB.

(I'm putting myself as an author on this ticket because I seem to be listed as one the patch...)

comment:28 Changed 11 years ago by Leif Leonhardy

Description: modified (diff)

comment:29 Changed 11 years ago by Leif Leonhardy

Merged in: sage-4.7.2.alpha3
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.