Add ccache as an optional spkg

[ohanar]

This is needed to eliminate the majority of the build time when switching between various branches in the git based workflow. It is also useful in the short run for dramatically speeding up the build process for many spkgs.

Installation:

• Add the spkg ​http://wstein.org/home/ohanar/spkgs/ccache-3.1.8.spkg to spkg/standard
• Apply trac13032_root.1.patch​ to the root repository
• Apply trac13032_scripts.patch​ to the scripts repository
• Apply trac13032_doc.patch​ to the sage library

[ohanar]

Not yet ready, the build scripts still need to be modified to set CC and CXX once this is installed.

[ppurka]

Two general comments:

1. We probably need to set the max cache size. On my system the default is 1G, which some people may dislike. It can be set using ccache -M <size>{G,M,K}, where the letters G, M, K can be used to specify the usual gigabytes, megabytes and kilobytes. This needs to be set only if CCACHE_DIR is empty, since the user may have his/her own ccache settings.
2. This fact about the ccache size should be mentioned in the documentation, so that the user knows how much of disk space will be used.

[ohanar]

Replying to ppurka:

Two general comments:

1. We probably need to set the max cache size. On my system the default is 1G, which some people may dislike. It can be set using ccache -M <size>{G,M,K}, where the letters G, M, K can be used to specify the usual gigabytes, megabytes and kilobytes.

This is set in the spkg-install to 3G.

This needs to be set only if CCACHE_DIR is empty, since the user may have his/her own ccache settings.

2. This fact about the ccache size should be mentioned in the documentation, so that the user knows how much of disk space will be used.

Both good points, I'll see about changing these.

[ohanar]

1. The f90cache source code does not include a copy of its license. Since it is based off of and old version of ccache, it must be GPL licensed, so it is in violation of GPL license. For this reason, I'm going to remove it from this ticket.
2. ccache depends on zlib and includes its own copy to use if it can't find one. I've updated the spkg and build scripts to ensure that it uses our copy.

[ohanar]

apply to scripts repo

[kini]

Have you reported the f90cache licensing problem upstream, or asked for a special dispensation?

[ohanar]

Replying to kini:

Have you reported the f90cache licensing problem upstream, or asked for a special dispensation?

I just reported the problem upstream. A special dispensation is irrelevant, it must be GPL since it derivative of ccache-2.4, which is licensed under GPL-2+, so it is an issue with violating the GPL.

Regardless, I think it would be best to place a f90cache spkg in a different ticket.

[kini]

AFAIK the licensing classification of software applies not to the body of its source code in general but to a particular act of distribution of that body of source code (though open source projects often offer public downloads of their source code with the understanding that each download of the code is taken to be governed by the license specified in the COPYING file and referred to as the license "of the software" itself). By "special dispensation" I meant the obtainment of a copy of the source code from the copyright holder under a one-time transferral stipulated to be under GPLv2+, even while the public download link was perhaps not governed by GPLv2+.

[jdemeyer]

Replying to kini:

AFAIK the licensing classification of software applies not to the body of its source code in general but to a particular act of distribution of that body of source code (though open source projects often offer public downloads of their source code with the understanding that each download of the code is taken to be governed by the license specified in the COPYING file and referred to as the license "of the software" itself). By "special dispensation" I meant the obtainment of a copy of the source code from the copyright holder under a one-time transferral stipulated to be under GPLv2+, even while the public download link was perhaps not governed by GPLv2+.

I don't understand what you're trying to say here. The GPL talks both about running the code (I guess this is what you refer to when talking about "the body of its source code in general") and distributing the code. Running and distributing are two essentially independent things, but both governed by the same license.

[jdemeyer]

In the doc patch: the command

sage -sh ccache --max-size=SIZE

surely isn't correct. Either you mean "first run ./sage --sh, then ccache --max-size=SIZE", or you mean "sage --sh -c "ccache --max-size=SIZE"" or you mean "sage --ccache --max-size=SIZE" with adding a --ccache option to spkg/bin/sage.

[kini]

I mean that when you say "Software X is under the GPL", what you actually mean is "when I obtained software X, I did so under the GPL". If I am the copyright holder to software X, there is no contradiction in me giving it to person A under the GPL and person B under the BSD license. Similarly, if I'm the copyright holder of f90cache, there's no contradiction if I give f90cache to J. Random Hacker under an ill-specified non-GPL license, but at the same time give f90cache to ohanar under GPLv2+.

Of course, it turns out I am actually contractually forbidden by the ccache authors from licensing f90cache to J. Random Hacker under this ill-specified non-GPL license, because f90cache is a derivative work of a version of ccache which I myself obtained under the GPLv2+. But there's nothing wrong with my simultaneous licensing of f90cache to ohanar under GPLv2+. At least, that's my understanding of it, based on what guys on #fsf told me a long time ago - IANAL, needless to say :)

[ohanar]

Replying to jdemeyer:

In the doc patch: the command

sage -sh ccache --max-size=SIZE

surely isn't correct.

You are right, I could have sworn that worked. Updating the patch now...

[ohanar]

Updated to 3.1.8 and removed fuzz. Otherwise, everything still works fine...

[jdemeyer]

If ccache is only installed conditionally, why not make it an optional package?

[ppurka]

Replying to jdemeyer:

If ccache is only installed conditionally, why not make it an optional package?

That's quite true. Maybe the ccache package can be downloaded automatically if the user sets the ccache flags in the environment. This will ensure that the distributed Sage sources does not increase in size.

[ohanar]

We can make it optional -- the only changes necessary would be in the merging process (just don't bundle the spkg) and the documentation. The reason I was making it standard was because it will be needed for fast recompiles when switching branches in got, but for now we don't do branching, so it doesn't need to be standard.

[ohanar]

apply to sage library

[ohanar]

Ok, updated the documentation.

[ohanar]

apply to root repo

[ohanar]

Actually it turns out you have to make a bit of a hack to make it optional while integrating with the build system (since part of the point is that you would want it to cache as much of the build as possible).

I've been working with the git layout too much, where it is easy to get to the point of newest_version working, but where sage-spkg can't find the spkgs, so it tries to download them all (problem here is that newest_version relies on the spkg being included with the sources).

[ppurka]

It's working nice. I downloaded 5.4beta1 and started the compilation with ccache enabled. Then I canceled the compilation after some time and ran make distclean and restarted all over again. Stuff that was previously compiled was flying by really fast. :)

I will have to see how much it improves when there is an upgrade.

[ppurka]

A general question. How can I upgrade my installation with this patch in place? Trying to upgrade modifies some files in SAGE_ROOT and then the upgrade fails with the following message

Successfully installed pynac-0.2.5
Deleting temporary build directory
/home/punarbasu/Installations/sage-5.4.beta1/spkg/build/pynac-0.2.5
Making Python scripts relocatable...
Finished installing pynac-0.2.5.spkg
make[1]: Leaving directory `/home/punarbasu/Installations/sage-5.4.beta1/spkg'
make: *** [all] Error 2

real    0m49.350s
user    0m59.432s
sys     0m7.064s
Error building Sage.
Error installing Sage!
Double checking that all packages have been installed.
Downloading packages from 'http://jambu.spms.ntu.edu.sg/sage/devel/sage-5.4.rc1/spkg'.
Reading package lists...  Done!
The following packages will be upgraded:

    conway_polynomials-0.3 elliptic_curves-0.7 extcode-5.4.rc1
    gcc-4.6.3 graphs-20120404.p4 polytopes_db-20100210.p2 r-2.14.0.p6
    sage-5.4.rc1 sage_root-5.4.rc1 sage_scripts-5.4.rc1

 ** WARNING: This is a source-based upgrade, which could take hours,
 ** fail, and render your Sage install useless!!

Do you want to continue [y/N]? y
There are uncommitted changes in the Sage root repository:
M spkg/install
M spkg/standard/deps

Aborting.

[ppurka]

Unfortunately, ccache doesn't help very much in the compilation of a new Sage tarball. Most of the time is taken up by ATLAS. I have a failed build of 5.4.rc4 here that failed at ATLAS, after a total compilation time of just over 3 hours.

real	187m47.272s
user	236m57.749s
sys	12m48.248s
Error building Sage.
make: *** [build] Error 1
  14218.03s user 771.96s system 133% cpu 3:07:48.73 total

EDIT: In comparison, a successful build of sage (make build) takes just over 4 hours.

[ohanar]

I never build ATLAS, so I can't comment on how well ccache works with it, but I don't really think it matters much since everyone who would gain any sort of benefit from ccache should already be using SAGE_ATLAS_LIB. I do know that R seems to not get much out of ccache though.

[ppurka]

I am installing a system ATLAS now. I have been bitten a bit too many times with failing ATLAS builds with sage. It's not a fault of sage; it is probably that the laptop overheats or something. It builds fine on a desktop.

That said, the root patch needs to be rebased on sage-5.4.rc. The rebased patch is ​available here. I forgot to upload it, but maybe you can take that, rename it, and upload it here, as the original file name.

[robertwb]

Replying to ppurka:

Unfortunately, ccache doesn't help very much in the compilation of a new Sage tarball. Most of the time is taken up by ATLAS. I have a failed build of 5.4.rc4 here that failed at ATLAS, after a total compilation time of just over 3 hours.

real	187m47.272s
user	236m57.749s
sys	12m48.248s
Error building Sage.
make: *** [build] Error 1
  14218.03s user 771.96s system 133% cpu 3:07:48.73 total

EDIT: In comparison, a successful build of sage (make build) takes just over 4 hours.

Just to check, cchache is shared across sage builds, right?

Regarding ATLAS, ideally most people will either be using a system build or match one of the pre-shipped tuning parameters.

[ppurka]

Replying to robertwb:

Just to check, cchache is shared across sage builds, right?

Yes. It is shared across different installations.

[ppurka]

Apply to SAGE_ROOT

[ppurka]

This has been working fine for me for several months already. Setting it to positive review.

I updated the root patch to apply to more recent sage (since sage-5.5beta onwards).

[kini]

Thanks, Basu!

[jdemeyer]

Why this:

if [ -z "$CCACHE_DIR" ]; then 
    CCACHE_DIR="$DOT_SAGE/ccache" 
fi 
if [ -z "$CYCACHE_DIR" ]; then 
    CYCACHE_DIR="$DOT_SAGE/cycache" 
fi 

Why not use the default directories?

[ppurka]

Replying to jdemeyer:

Why not use the default directories?

Because it keeps the sage files in a fixed sage directory instead of the home directory of the user. The default sizes of those directories are huge (4G) and are needed because Sage itself is huge. I am already using 3.5G of that.

~/.sage/ccache» du -sh .
3.5G	.

[schilly]

ccache spkg is on the server

[jdemeyer]

I still don't quite like how this patch uses a separate CCACHE_DIR inside .sage. Because now, I have:

$ ccache -s
cache directory                     /home/jdemeyer/.ccache
[...]
cache size                           3.2 Gbytes
max cache size                       4.0 Gbytes

$ ./sage --sh
(sage-sh) ccache -s
cache directory                     /home/jdemeyer/.sage//ccache
[...]
cache size                         935.1 Mbytes
max cache size                       1.0 Gbytes

I would like to have just one .ccache directory, especially since my default ccache dir is already used for Sage. I don't think we should add 1GB of data inside the home directories of unsuspecting users.

Also note that the default size of 1GB is too little anyway to be useful for Sage.

[ppurka]

You can of course have only one ccache dir. In that case, simply define the environment variable CCACHE_DIR in your shell rc file. Then everything will use that directory. The default is set for those people who don't bother with setting up ccache variables.

~» sage --sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/punarbasu/Installations/sage-5.6.beta1
~» ccache -s
cache directory                     /home/punarbasu/.sage//ccache
cache hit (direct)                  1620
cache hit (preprocessed)           13825
cache miss                         16881
called for link                     3066
called for preprocessing            1553
multiple source files                 20
compiler produced stdout               2
compile failed                       389
preprocessor error                   236
bad compiler arguments               285
unsupported source language         1214
autoconf compile/link               3228
unsupported compiler option          786
no input file                       1345
files in cache                     53031
cache size                           2.7 Gbytes
max cache size                       4.0 Gbytes
~»
Exited Sage subshell.



~» export CCACHE_DIR=$HOME/.ccache
~» sage --sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/punarbasu/Installations/sage-5.6.beta1
~» ccache -s
cache directory                     /home/punarbasu/.ccache
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
files in cache                         0
cache size                             0 Kbytes
max cache size                       1.0 Gbytes

[jdemeyer]

Replying to ppurka:

The default is set for those people who don't bother with setting up ccache variables.

Not really. I didn't bother setting CCACHE_DIR (since I like the default) but I don't like this ticket's defaults. It makes it harder to configure ccache for example (it needs to be done inside a Sage shell) and I still don't understand why somebody would want a separate ccache directory for Sage.

[jhpalmieri]

Maybe there should be a new environment variable SAGE_CCACHE_DIR. If it's not set (the default), then use CCACHE_DIR. Otherwise, use it, in case someone wants to use a separate ccache directory for Sage.

ppurka, you also didn't address the comment about 1GB being "too little anyway to be useful for Sage".

[jdemeyer]

I don't think we should introduce yet another environment variable. Users wishing to set a Sage-only ccache dir can use ​.sage/sagerc

[ppurka]

I don't understand what the confusion is about. There are only two possibilities.

1. If you set CCACHE_DIR in your shell's rc file, then that variable will be used and none of your ccache settings will be changed. Here is an example on a test account where I exported the setting CCACHE_DIR=~/.ccache explicitly before compiling sage. Then I stopped the compilation of sage midway and looked at the output of ccache -s from a different terminal. All of this is outside of sage shell.

   test@ub2 ~ $ ccache -s
   cache directory                     /tmp/test/.ccache
   cache hit (direct)                     1
   cache hit (preprocessed)               2
   cache miss                           139
   called for link                        3
   called for preprocessing              46
   compile failed                         5
   preprocessor error                     5
   bad compiler arguments                 4
   autoconf compile/link                 75
   no input file                         18
   files in cache                       288
   cache size                           3.6 Mbytes
   max cache size                       1.0 Gbytes
   

2. If you do not provide a ccache setting at all, then the package sets the ccache size and the ccache directory to some defaults. Since, you are not showing any preference here, the ccache directory is set to ~/.sage/ccache and the size is set to 4G. All sage related stuff is in one single directory ~/.sage by default - I don't understand why we should go out of our way to mix it with user's own settings in their home directories.

[jdemeyer]

Replying to ppurka:

If you do not provide a ccache setting at all [...] the size is set to 4G.

That's already false, ​the default is 1GB.

All sage related stuff is in one single directory ~/.sage by default - I don't understand why we should go out of our way to mix it with user's own settings in their home directories.

Because I don't consider cache compilation files to be Sage-related.

Imagine I want to compile a package X outside of Sage, a package which happens to be part of Sage (I do that with PARI for example). Then it's a pity that the cached object files for package X appear twice in my home directory: once in .ccache and once in .sage.

My problem really boils down to this: why would you want a separate ccache directory for Sage?

[jdemeyer]

Replying to ppurka:

Since, you are not showing any preference here

I would say that, by not setting CCACHE_DIR, I am showing a preference for the default ccache directory $HOME/.ccache

I think that setting a variable explicitly to its default value should be the same as not setting it at all.

[ppurka]

Replying to jdemeyer:

Replying to ppurka:

If you do not provide a ccache setting at all [...] the size is set to 4G.

That's already false, ​the default is 1GB.

I must reiterate. If you don't provide any ccache directory settings at all, then it is set to 4GB​ with this ticket/spkg.

Imagine I want to compile a package X outside of Sage, a package which happens to be part of Sage (I do that with PARI for example). Then it's a pity that the cached object files for package X appear twice in my home directory: once in .ccache and once in .sage.

Well, then you set your ccache environment variable appropriately. To use ccache while compiling with sage you need to set SAGE_INSTALL_CCACHE=yes in your environment variable. Then, it will use whatever ccache directory you have set and otherwise it will use the default of ~/.sage/ccache.

My problem really boils down to this: why would you want a separate ccache directory for Sage?

In fact, one question that I would ask is - what other sage setting is present outside the ~/.sage directory? If there is, then it makes sense to mix and match sage specific settings and folders with system/user specific settings.

[kini]

You might want to put this question to sage-devel if you haven't already. I think there might be people on both sides of the argument.

On the one hand I agree that setting a variable explicitly to its default value should be the same as not setting it at all. On the other hand, I feel like I'd want a separate ccache directory for Sage because I would like all Sage artifacts to be segregated from other stuff on my system, as they currently are by default. Currently, Sage will never modify anything outside its directory except /tmp and $DOT_SAGE unless you specifically write code that does so (right?). Not having a separate Sage ccache directory would add ~/.ccache to the list.

A compromise/workaround would be to, say, symlink ~/.sage/ccache to ~/.ccache manually if you want to share caches. Is this too much of a burden, do you think, Jeroen? The SPKG installer could print a message about this possibility, or something.

[jdemeyer]

Replying to ppurka:

I must reiterate. If you don't provide any ccache directory settings at all, then it is set to 4GB​ with this ticket/spkg.

OK, so it's something specific to this spkg. When using system ccache, I still get 1GB of cache inside .sage, even if I manually configured ccache to use 4GB.

To use ccache while compiling with sage you need to set SAGE_INSTALL_CCACHE=yes in your environment variable.

Again false, as a system ccache might be used.

In fact, one question that I would ask is - what other sage setting is present outside the ~/.sage directory? If there is, then it makes sense to mix and match sage specific settings and folders with system/user specific settings.

Probably Sage's readline uses non-Sage-specific configuration files ($HOME/.inputrc).

[jdemeyer]

Replying to kini:

A compromise/workaround would be to, say, symlink ~/.sage/ccache to ~/.ccache manually if you want to share caches. Is this too much of a burden, do you think, Jeroen? The SPKG installer could print a message about this possibility, or something.

Of course it's not much burden for me, but what about other people in the same situation? This patch changes the ccache directory and the cache size behind people's back if they were using system ccache before. So this patch can make performance strictly worse without users being aware of it.

[jdemeyer]

Replying to kini:

Not having a separate Sage ccache directory would add ~/.ccache to the list.

Which is a problem because ...........?

[kini]

Probably Sage's readline uses non-Sage-specific configuration files ($HOME/.inputrc).

Does Sage change ~/.inputrc or just read it? Sage of course reads tons of files all over the system. An example of another dotfile in ~ that Sage (or Sage's Mercurial anyway) reads is ~/.hgrc. But it will change ~/.ccache.

Of course it's not much burden for me, but what about other people in the same situation?

As I said, we can print a note at the end of the SPKG installation process, or otherwise document it. I suspect that most users will not mind that the ccache directory ~/.sage/ccache is used, and if they do, it is probably because they are already running ccache on their system and want to consolidate the cache directories together, so they're probably savvy enough to make a symlink.

This patch changes the ccache directory and the cache size behind people's back if they were using system ccache before.

Sage installs another copy of Python on the system "behind people's back" as well and nobody complains (or at least we don't listen to such complaints). I don't see the problem.

Also, maybe I am misunderstanding something. Are you saying that installing this SPKG into Sage will change the behavior of the system ccache running independently of Sage?

Which is a problem because ...........?

For one thing, it means more detritus around the system to clean up if you want to remove Sage from your system. And very large unwanted detritus at that, if the user doesn't have a system ccache. Of course, it's an optional package, so we can assume the user at least knows they have installed the ccache SPKG, but most users would expect that they can completely eradicate Sage and all its artifacts by deleting the Sage directory and $DOT_SAGE.

Also, this kind of lack of compartmentalization just seems unhygienic. I kind of don't like the idea of Sage interacting with and modifying data that belongs to other programs which I've installed systemwide. I like that Sage is a sandbox mini-distribution of software which is self-contained. (Or, at least that's the paradigm the official Sage distribution has conformed to so far, sage-on-gentoo and sage-on-debian notwithstanding.)

[jdemeyer]

Replying to kini:

Also, maybe I am misunderstanding something. Are you saying that installing this SPKG into Sage will change the behavior of the system ccache running independently of Sage?

No, that's not the case at all. The case I am considering is the opposite: not installing this SPKG into Sage but using a system-wide ccache.

[jdemeyer]

Here is a proposal for a compromise: use $HOME/.ccache if using a system-wide ccache installation and use $DOT_SAGE/ccache if using Sage's ccache, which we test by

if [ -x "$SAGE_LOCAL/bin/ccache" ];

[jdemeyer]

Replying to kini:

I kind of don't like the idea of Sage interacting with and modifying data that belongs to other programs which I've installed systemwide.

I'd argue that in this case, the interaction is a feature, not a bug.

And very large unwanted detritus at that, if the user doesn't have a system ccache

But I do have system-wide ccache. For me, this patch adds 1GB of extra detritus in .sage/ccache and actually reduces performance by using a smaller cache.

[ppurka]

I think what you are really asking about is

What should the zero-configuration scenario, even for advanced users, look like?

I count you as an advanced user since you are using a user-wide ccache, and you don't want sage to use any other ccache directory by default. On top of that, you want sage to use the ccache directory that you are already using, which may or may not be the default ccache directory.

I think this question should be asked to the general audience in sage-devel. There might be people who keep the ~/.sage directory separate, along with the rest of their sage installation. For a heavy user of sage or for someone running a server, I can contemplate that the ~/.sage (or DOT_SAGE) will not be small and there might be people who are using a different (maybe mounted) directory that is separate from their home directory.

[jdemeyer]

Replying to ppurka:

I think this question should be asked to the general audience in sage-devel.

Done.

[jhpalmieri]

Since the spkg is pretty small, would it make sense to distribute it with Sage? It would still only be built if SAGE_INSTALL_CCACHE were set, so it would still be in some sense optional. But then one wouldn't need internet access when building Sage, and the install log for ccache would have a more informative name: it would include the version number.

[jdemeyer]

Replying to jhpalmieri:

Since the spkg is pretty small, would it make sense to distribute it with Sage? It would still only be built if SAGE_INSTALL_CCACHE were set, so it would still be in some sense optional.

I don't mind.

[jdemeyer]

I created a patch at #13938 to not set CCACHE_DIR, needs review.