Opened 9 years ago
Closed 8 years ago
#10801 closed enhancement (fixed)
Create a new option: "sage -strip" which deletes things that aren't needed for a binary distribution of sage, or for people that will never develop or upgrade
Reported by: | was | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-4.7.2 |
Component: | packages: standard | Keywords: | sd32 |
Cc: | jason, kcrisman | Merged in: | sage-4.7.2.alpha3 |
Authors: | William Stein, Keshav Kini | Reviewers: | Benjamin Jones, Keshav Kini |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
This is a frequently requested feature. We should start with some little script that does this, and build on it. I tried all the following, and ran the test suite and it worked fine.
- rm SAGE_ROOT/local/lib/*.a
- rm SAGE_ROOT/devel/sage/build/lib.*
- rm SAGE_ROOT/devel/sage/build/temp.*
- strip SAGE_ROOT/local/bin/Singular-* SAGE_ROOT/local/bin/gfan # gfan is a huge win.
- jsmath image fonts are in MoinMoin? and are *HUGE*. Just delete everything related to moinmoin...
- strip SAGE_ROOT/local/lib/*.so
- rm -rf SAGE_ROOT/local/python/site-package/MoinMoin
- rm all files in SAGE_ROOT/devel/sage/sage/ that begin "* Generated by Cython" (I didn't do that)
The patches below actually provide a new make
target, "micro_release
", rather than adding a new option to sage
.
Apply trac_10801-root_repo.patch to the Sage root repository.
Apply trac_10801-local_bin_repo.patch to the Sage scripts repository.
Attachments (2)
Change History (31)
comment:1 Changed 9 years ago by
- Cc jason added
comment:2 Changed 9 years ago by
If we mount a sage source installation with mount option strictatime or relatime we can just touch all the files to have an access time in the deep past (with something like cd $SAGE_ROOT ; for x in $(find) ; do touch -ad "1981-02-03 04:05:06" ; done
), then run the whole doctest suite, then use stat() to find files which were unused. This should give us a good idea of what is never used. Unfortunately this does not seem possible on sage.math - POSIX atime is not respected in my homedir, probably as it is mounted over NFS...
comment:3 Changed 9 years ago by
- Description modified (diff)
comment:4 Changed 9 years ago by
if you're using linux, take a look at inotify-tools which has nice and clean command line tools, as it will give you a more fine-grained and realtime sense of what files are being used as they are accessed/modified to track down what's being used by which calls/tests/parts of sage.
comment:5 Changed 9 years ago by
We can probably remove any Mercurial repositories.
comment:6 Changed 9 years ago by
- Cc kcrisman added
comment:7 Changed 9 years ago by
Here's some data I collected about what files are used and what files aren't, in sage 4.7.alpha3. Surprisingly many aren't, but come to think of it there's going to be a lot of stuff that users would want which isn't used by any doctest. I guess one should think of this data as a lower bound...
comment:8 follow-up: ↓ 12 Changed 8 years ago by
Why is it OK to delete SAGE_ROOT/local/bin/gfan?
comment:9 Changed 8 years ago by
Not delete, but strip. This removes debug symbols.
comment:10 Changed 8 years ago by
Slightly related: sage -crap ...
(local/bin/sage-crap
)
Note that we already have a few *clean
targets in the top-level Makefile; there could be more (like strip
), and some of the others could perhaps get improved or extended.
I personally wouldn't use -strip
for anything else than really stripping debug symbols, i.e., if it is e.g. intended to delete also static libraries (which may be needed if someone installs some package, not necessarily an spkg), I'd choose a different name for such an option (or make
target).
comment:11 Changed 8 years ago by
After a long discussion, Keshav and I came up with:
make micro_release
since this is a lot like "make release" with other software....
comment:12 in reply to: ↑ 8 Changed 8 years ago by
Replying to mhampton:
Why is it OK to delete SAGE_ROOT/local/bin/gfan?
I didn't suggest deleting gfan, but *stripping* it.
comment:13 Changed 8 years ago by
Here's how I'm testing this:
wstein@sage:/tmp/wstein/sage-4.7.2.alpha1-sage.math.washington.edu-x86_64-Linux$ $ du -sh . 2.4G . $ du -s . 2468404 . $ ./sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-root_repo.patch $ cd local/bin/; ../../sage -hg import http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch applying http://trac.sagemath.org/sage_trac/raw-attachment/ticket/10801/trac_10801-local_bin_repo.patch $ time make micro_release real 0m38.227s user 0m2.370s sys 0m5.140s $ du -sh . 1.5G . $ du -s . 1547480 . $ ./sage # see it works $ make ptestlong # observe what happens (I will report back in a follow comment about this).
comment:15 follow-up: ↓ 16 Changed 8 years ago by
Some thoughts:
- In
strip_binaries()
, why not strip all executables? One can find them by checking exit status offind . -perm /111 | grep "not stripped" &> /dev/null
, I think. - The *
_with_condition()
functions should not talk about extensions in their docstrings. - What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)
Going a bit farther, considering the target users will never develop or upgrade,
- Should we delete mercurial repos?
- Should we delete SPKGs?
comment:16 in reply to: ↑ 15 Changed 8 years ago by
Replying to kini:
Some thoughts:
- In
strip_binaries()
, why not strip all executables? One can find them by checking exit status offind . -perm /111 | grep "not stripped" &> /dev/null
, I think.
LATER: This could be added, but can we do it later in a future ticket. It would be good if the first version of this patch is quite safe. Hey, it already saves *1 GB* as is!
- The *
_with_condition()
functions should not talk about extensions in their docstrings.
OK: I'll fix the patch right now.
- What about getting rid of MoinMoin? (fake edit: never mind, the ticket description is wrong or out of date - it's under 20 MB)
NO: Pointless, and scary in that it would break functionality, which is probably a bad idea. Good point about the size. I must have been testing something with some optional fonts installed.
Going a bit farther, considering the target users will never develop or upgrade,
- Should we delete mercurial repos?
FUTURE: Yes, but in a future version of this?
- Should we delete SPKGs?
FUTURE: It's safer to put a small string in each, to avoid confusion. This happens automatically when you build a binary with "sage -bdist". Anyway, this is something that could be added later.
comment:17 Changed 8 years ago by
- Reviewers set to Benjamin Jones
Tested the patches on sage.math and fresh build of Sage-4.7.1:
bjones@sage:/scratch/bjones/sage-4.7.1$ pwd /scratch/bjones/sage-4.7.1 bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh 2.5G . bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s 2554916 .
bjones@sage:/scratch/bjones/sage-4.7.1$ time make micro_release ... real 0m12.064s user 0m2.560s sys 0m7.600s bjones@sage:/scratch/bjones/sage-4.7.1$ du . -sh 2.0G . bjones@sage:/scratch/bjones/sage-4.7.1$ du . -s 2033236 .
So it saved 0.5 GB which is not the 1.0 GB observed by William above. Did you remove stuff by hand in that session that the script doesn't touch? Or maybe because I built the install from source instead of using a binary distribution?
I'm running make ptestlong
now, will report back if anything fails.
comment:18 Changed 8 years ago by
After producing the micro release, make ptestlong
had one failure which disappeared when I doctested by hand:
The following tests failed: sage -t -long -force_lib devel/sage/sage/tests/startup.py # 1 doctests failed ---------------------------------------------------------------------- Total time for all tests: 1468.7 seconds make: *** [ptestlong] Error 128
Subsequently by hand:
bjones@sage:/scratch/bjones/sage-4.7.1$ sage -t -long -force_lib devel/sage/sage/tests/startup.py sage -t -long -force_lib "devel/sage/sage/tests/startup.py" [12.1 s] ---------------------------------------------------------------------- All tests passed! Total time for all tests: 12.1 seconds
comment:19 Changed 8 years ago by
Ben:
- Regarding the size savings, I started with a binary... so we may see a difference.
- Regarding the startup time failure, this has nothing to do with my patch. It routinely fails on sage.math. It's a test that it isn't taking more than 2 seconds on sage.math. I put it in back when we got the startup time down to solidly be < 1 second on sage.math, since I didn't want people to screw that up. They did though.
comment:20 Changed 8 years ago by
- Keywords sd32 added
comment:21 Changed 8 years ago by
- Reviewers changed from Benjamin Jones to Benjamin Jones, Keshav Kini
- Status changed from needs_review to positive_review
William: OK, sounds good. I'll make a ticket. Sounds like we're good to go here, no? Positive review from me.
comment:22 Changed 8 years ago by
OK, I checked on OSX 10.6.8 binary install and I get results simular to William's. Looks good here too. Positive review from me as well.
comment:23 Changed 8 years ago by
New ticket is #11743.
comment:24 Changed 8 years ago by
Martin Albrecht pointed out that deleting .a files could make it so users can't compile certain Cython programs, maybe. I'm not sure if we should care though, given the goal of this ticket ('never develop or upgrade').
comment:25 Changed 8 years ago by
Hm. If that includes %cython
cells, I don't think that counts as "developing".
comment:26 Changed 8 years ago by
Yes, I'm concerned. I'll simply comment out the line that deletes the .a files, and repost the patch.
comment:27 Changed 8 years ago by
Fair enough. Sadly this reduces our space savings by more than 200 MB.
(I'm putting myself as an author on this ticket because I seem to be listed as one the patch...)
comment:28 Changed 8 years ago by
- Description modified (diff)
comment:29 Changed 8 years ago by
- Merged in set to sage-4.7.2.alpha3
- Resolution set to fixed
- Status changed from positive_review to closed
It seems like the LiveCD folks have also worked on stuff like this...