Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade

[William Stein]

This is a frequently requested feature. We should start with some little script that does this, and build on it. I tried all the following, and ran the test suite and it worked fine.

• rm SAGE_ROOT/local/lib/*.a
• rm SAGE_ROOT/devel/sage/build/lib.*
• rm SAGE_ROOT/devel/sage/build/temp.*
• strip SAGE_ROOT/local/bin/Singular-* SAGE_ROOT/local/bin/gfan # gfan is a huge win.
• jsmath image fonts are in MoinMoin? and are *HUGE*. Just delete everything related to moinmoin...
• strip SAGE_ROOT/local/lib/*.so
• rm -rf SAGE_ROOT/local/python/site-package/MoinMoin
• rm all files in SAGE_ROOT/devel/sage/sage/ that begin "* Generated by Cython" (I didn't do that)

---

The patches below actually provide a new make target, "micro_release", rather than adding a new option to sage.

---

Apply trac_10801-root_repo.patch​ to the Sage root repository.

Apply trac_10801-local_bin_repo.patch​ to the Sage scripts repository.

[Jason Grout]

It seems like the LiveCD folks have also worked on stuff like this...

[Keshav Kini]

If we mount a sage source installation with mount option strictatime or relatime we can just touch all the files to have an access time in the deep past (with something like cd $SAGE_ROOT ; for x in $(find) ; do touch -ad "1981-02-03 04:05:06" ; done), then run the whole doctest suite, then use stat() to find files which were unused. This should give us a good idea of what is never used. Unfortunately this does not seem possible on sage.math - POSIX atime is not respected in my homedir, probably as it is mounted over NFS...

[Paul Ivanov]

if you're using linux, take a look at inotify-tools which has nice and clean command line tools, as it will give you a more fine-grained and realtime sense of what files are being used as they are accessed/modified to track down what's being used by which calls/tests/parts of sage.

[John Palmieri]

We can probably remove any Mercurial repositories.

[Keshav Kini]

​Here's some data I collected about what files are used and what files aren't, in sage 4.7.alpha3. Surprisingly many aren't, but come to think of it there's going to be a lot of stuff that users would want which isn't used by any doctest. I guess one should think of this data as a lower bound...

[mhampton]

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

[Keshav Kini]

Not delete, but ​strip. This removes debug symbols.

[Leif Leonhardy]

Slightly related: sage -crap ... (local/bin/sage-crap)

Note that we already have a few *clean targets in the top-level Makefile; there could be more (like strip), and some of the others could perhaps get improved or extended.

I personally wouldn't use -strip for anything else than really stripping debug symbols, i.e., if it is e.g. intended to delete also static libraries (which may be needed if someone installs some package, not necessarily an spkg), I'd choose a different name for such an option (or make target).

[William Stein]

After a long discussion, Keshav and I came up with:

   make micro_release

since this is a lot like "make release" with other software....

[William Stein]

Replying to mhampton:

Why is it OK to delete SAGE_ROOT/local/bin/gfan?

I didn't suggest deleting gfan, but *stripping* it.

[William Stein]

apply to the repo in $SAGE_ROOT

[William Stein]

Here's how I'm testing this:

wstein@sage:/tmp/wstein/sage-4.7.2.alpha1-sage.math.washington.edu-x86_64-Linux$
$ du -sh .
2.4G    .
$ du -s .
2468404 .

$ ./sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch
applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch

$ cd local/bin/; ../../sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch
applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch

$ time make micro_release
real    0m38.227s
user    0m2.370s
sys     0m5.140s

$ du -sh .
1.5G    .
$ du -s .
1547480 .

$ ./sage
# see it works

$ make ptestlong
# observe what happens (I will report back in a follow comment about this).

[William Stein]

ptestlong passes. W00t.

[Keshav Kini]

Some thoughts:

• In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.
• The *_with_condition() functions should not talk about extensions in their docstrings.
• What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

Going a bit farther, considering the target users will never develop or upgrade,

• Should we delete mercurial repos?
• Should we delete SPKGs?

[William Stein]

Replying to kini:

Some thoughts:

• In strip_binaries(), why not strip all executables? One can find them by checking exit status of find . -perm /111 | grep "not stripped" &> /dev/null, I think.

LATER: This could be added, but can we do it later in a future ticket. It would be good if the first version of this patch is quite safe. Hey, it already saves *1 GB* as is!

• The *_with_condition() functions should not talk about extensions in their docstrings.

OK: I'll fix the patch right now.

• What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)

NO: Pointless, and scary in that it would break functionality, which is probably a bad idea. Good point about the size. I must have been testing something with some optional fonts installed.

Going a bit farther, considering the target users will never develop or upgrade,

• Should we delete mercurial repos?

FUTURE: Yes, but in a future version of this?

• Should we delete SPKGs?

FUTURE: It's safer to put a small string in each, to avoid confusion. This happens automatically when you build a binary with "sage -bdist". Anyway, this is something that could be added later.

[Benjamin Jones]

Tested the patches on sage.math and fresh build of Sage-4.7.1:

bjones@sage:/scratch/bjones/sage-4.7.1$ pwd
/scratch/bjones/sage-4.7.1
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.5G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2554916	.

bjones@sage:/scratch/bjones/sage-4.7.1$ time make micro_release
...
real	0m12.064s
user	0m2.560s
sys	0m7.600s
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh
2.0G	.
bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s
2033236	.

So it saved 0.5 GB which is not the 1.0 GB observed by William above. Did you remove stuff by hand in that session that the script doesn't touch? Or maybe because I built the install from source instead of using a binary distribution?

I'm running make ptestlong now, will report back if anything fails.

[Benjamin Jones]

After producing the micro release, make ptestlong had one failure which disappeared when I doctested by hand:

The following tests failed:

        sage -t  -long -force_lib devel/sage/sage/tests/startup.py # 1 doctests failed
----------------------------------------------------------------------
Total time for all tests: 1468.7 seconds
make: *** [ptestlong] Error 128

Subsequently by hand:

bjones@sage:/scratch/bjones/sage-4.7.1$ sage -t  -long -force_lib devel/sage/sage/tests/startup.py
sage -t -long -force_lib "devel/sage/sage/tests/startup.py" 
         [12.1 s]
 
----------------------------------------------------------------------
All tests passed!
Total time for all tests: 12.1 seconds

[William Stein]

Ben:

1. Regarding the size savings, I started with a binary... so we may see a difference.

2. Regarding the startup time failure, this has nothing to do with my patch. It routinely fails on sage.math. It's a test that it isn't taking more than 2 seconds on sage.math. I put it in back when we got the startup time down to solidly be < 1 second on sage.math, since I didn't want people to screw that up. They did though.

[Keshav Kini]

William: OK, sounds good. I'll make a ticket. Sounds like we're good to go here, no? Positive review from me.

[Benjamin Jones]

OK, I checked on OSX 10.6.8 binary install and I get results simular to William's. Looks good here too. Positive review from me as well.

[Keshav Kini]

New ticket is #11743.

[William Stein]

Martin Albrecht pointed out that deleting .a files could make it so users can't compile certain Cython programs, maybe. I'm not sure if we should care though, given the goal of this ticket ('never develop or upgrade').

[Keshav Kini]

Hm. If that includes %cython cells, I don't think that counts as "developing".

[William Stein]

Yes, I'm concerned. I'll simply comment out the line that deletes the .a files, and repost the patch.

[William Stein]

replacing the patch by one that doesn't delete .a files.

[Keshav Kini]

Fair enough. Sadly this reduces our space savings by more than 200 MB.

(I'm putting myself as an author on this ticket because I seem to be listed as one the patch...)