Opened 10 years ago

Last modified 9 years ago

#13032 closed enhancement

Add ccache as an optional spkg — at Version 42

Reported by: ohanar Owned by: GeorgSWeber
Priority: major Milestone: sage-5.6
Component: packages: optional Keywords: sd40.5
Cc: kini, robertwb, ppurka, jdemeyer Merged in:
Authors: R. Andrew Ohana Reviewers:
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by ppurka)

This is needed to eliminate the majority of the build time when switching between various branches in the git based workflow. It is also useful in the short run for dramatically speeding up the build process for many spkgs.

Installation:

Change History (46)

comment:1 Changed 10 years ago by ohanar

  • Description modified (diff)

Not yet ready, the build scripts still need to be modified to set CC and CXX once this is installed.

comment:2 Changed 10 years ago by kini

  • Cc kini added

comment:3 Changed 10 years ago by ohanar

  • Dependencies set to #13040
  • Keywords sd40.5 added

comment:4 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:5 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:6 Changed 10 years ago by ohanar

  • Description modified (diff)
  • Summary changed from Add ccache as a standard spkg to Add ccache and f90cache as standard spkgs

comment:7 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:8 Changed 10 years ago by ohanar

  • Dependencies changed from #13040 to #13040 #13044
  • Description modified (diff)

comment:9 Changed 10 years ago by ohanar

  • Status changed from new to needs_review

comment:10 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:11 Changed 10 years ago by ohanar

  • Cc robertwb added

comment:12 Changed 10 years ago by ppurka

  • Cc ppurka added

comment:13 follow-up: Changed 10 years ago by ppurka

Two general comments:

  1. We probably need to set the max cache size. On my system the default is 1G, which some people may dislike. It can be set using ccache -M <size>{G,M,K}, where the letters G, M, K can be used to specify the usual gigabytes, megabytes and kilobytes. This needs to be set only if CCACHE_DIR is empty, since the user may have his/her own ccache settings.
  2. This fact about the ccache size should be mentioned in the documentation, so that the user knows how much of disk space will be used.

comment:14 in reply to: ↑ 13 Changed 10 years ago by ohanar

Replying to ppurka:

Two general comments:

  1. We probably need to set the max cache size. On my system the default is 1G, which some people may dislike. It can be set using ccache -M <size>{G,M,K}, where the letters G, M, K can be used to specify the usual gigabytes, megabytes and kilobytes.

This is set in the spkg-install to 3G.

This needs to be set only if CCACHE_DIR is empty, since the user may have his/her own ccache settings.

  1. This fact about the ccache size should be mentioned in the documentation, so that the user knows how much of disk space will be used.

Both good points, I'll see about changing these.

comment:15 Changed 10 years ago by ohanar

  • Status changed from needs_review to needs_work

comment:16 Changed 10 years ago by ohanar

  • Status changed from needs_work to needs_review

comment:17 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:18 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:19 Changed 10 years ago by ohanar

  • Description modified (diff)

comment:20 Changed 10 years ago by ohanar

  • Dependencies #13040 #13044 deleted

comment:21 Changed 10 years ago by ohanar

  • Cc jdemeyer added
  • Description modified (diff)
  • Summary changed from Add ccache and f90cache as standard spkgs to Add ccache as a standard spkg
  1. The f90cache source code does not include a copy of its license. Since it is based off of and old version of ccache, it must be GPL licensed, so it is in violation of GPL license. For this reason, I'm going to remove it from this ticket.
  2. ccache depends on zlib and includes its own copy to use if it can't find one. I've updated the spkg and build scripts to ensure that it uses our copy.

Changed 10 years ago by ohanar

apply to scripts repo

comment:22 follow-up: Changed 10 years ago by kini

Have you reported the f90cache licensing problem upstream, or asked for a special dispensation?

comment:23 in reply to: ↑ 22 Changed 10 years ago by ohanar

Replying to kini:

Have you reported the f90cache licensing problem upstream, or asked for a special dispensation?

I just reported the problem upstream. A special dispensation is irrelevant, it must be GPL since it derivative of ccache-2.4, which is licensed under GPL-2+, so it is an issue with violating the GPL.

Regardless, I think it would be best to place a f90cache spkg in a different ticket.

comment:24 follow-up: Changed 10 years ago by kini

AFAIK the licensing classification of software applies not to the body of its source code in general but to a particular act of distribution of that body of source code (though open source projects often offer public downloads of their source code with the understanding that each download of the code is taken to be governed by the license specified in the COPYING file and referred to as the license "of the software" itself). By "special dispensation" I meant the obtainment of a copy of the source code from the copyright holder under a one-time transferral stipulated to be under GPLv2+, even while the public download link was perhaps not governed by GPLv2+.

comment:25 in reply to: ↑ 24 Changed 10 years ago by jdemeyer

Replying to kini:

AFAIK the licensing classification of software applies not to the body of its source code in general but to a particular act of distribution of that body of source code (though open source projects often offer public downloads of their source code with the understanding that each download of the code is taken to be governed by the license specified in the COPYING file and referred to as the license "of the software" itself). By "special dispensation" I meant the obtainment of a copy of the source code from the copyright holder under a one-time transferral stipulated to be under GPLv2+, even while the public download link was perhaps not governed by GPLv2+.

I don't understand what you're trying to say here. The GPL talks both about running the code (I guess this is what you refer to when talking about "the body of its source code in general") and distributing the code. Running and distributing are two essentially independent things, but both governed by the same license.

comment:26 follow-up: Changed 10 years ago by jdemeyer

In the doc patch: the command

sage -sh ccache --max-size=SIZE

surely isn't correct. Either you mean "first run ./sage --sh, then ccache --max-size=SIZE", or you mean "sage --sh -c "ccache --max-size=SIZE"" or you mean "sage --ccache --max-size=SIZE" with adding a --ccache option to spkg/bin/sage.

comment:27 Changed 10 years ago by kini

I mean that when you say "Software X is under the GPL", what you actually mean is "when I obtained software X, I did so under the GPL". If I am the copyright holder to software X, there is no contradiction in me giving it to person A under the GPL and person B under the BSD license. Similarly, if I'm the copyright holder of f90cache, there's no contradiction if I give f90cache to J. Random Hacker under an ill-specified non-GPL license, but at the same time give f90cache to ohanar under GPLv2+.

Of course, it turns out I am actually contractually forbidden by the ccache authors from licensing f90cache to J. Random Hacker under this ill-specified non-GPL license, because f90cache is a derivative work of a version of ccache which I myself obtained under the GPLv2+. But there's nothing wrong with my simultaneous licensing of f90cache to ohanar under GPLv2+. At least, that's my understanding of it, based on what guys on #fsf told me a long time ago - IANAL, needless to say :)

comment:28 in reply to: ↑ 26 Changed 10 years ago by ohanar

Replying to jdemeyer:

In the doc patch: the command

sage -sh ccache --max-size=SIZE

surely isn't correct.

You are right, I could have sworn that worked. Updating the patch now...

comment:29 Changed 10 years ago by ohanar

  • Description modified (diff)

Updated to 3.1.8 and removed fuzz. Otherwise, everything still works fine...

comment:30 follow-up: Changed 10 years ago by jdemeyer

If ccache is only installed conditionally, why not make it an optional package?

comment:31 in reply to: ↑ 30 Changed 10 years ago by ppurka

Replying to jdemeyer:

If ccache is only installed conditionally, why not make it an optional package?

That's quite true. Maybe the ccache package can be downloaded automatically if the user sets the ccache flags in the environment. This will ensure that the distributed Sage sources does not increase in size.

comment:32 Changed 10 years ago by ohanar

  • Status changed from needs_review to needs_work
  • Summary changed from Add ccache as a standard spkg to Add ccache as an optional spkg

We can make it optional -- the only changes necessary would be in the merging process (just don't bundle the spkg) and the documentation. The reason I was making it standard was because it will be needed for fast recompiles when switching branches in got, but for now we don't do branching, so it doesn't need to be standard.

Last edited 10 years ago by ohanar (previous) (diff)

Changed 10 years ago by ohanar

apply to sage library

comment:33 Changed 10 years ago by ohanar

  • Status changed from needs_work to needs_review

Ok, updated the documentation.

Changed 10 years ago by ohanar

apply to root repo

comment:34 Changed 10 years ago by ohanar

Actually it turns out you have to make a bit of a hack to make it optional while integrating with the build system (since part of the point is that you would want it to cache as much of the build as possible).

I've been working with the git layout too much, where it is easy to get to the point of newest_version working, but where sage-spkg can't find the spkgs, so it tries to download them all (problem here is that newest_version relies on the spkg being included with the sources).

comment:35 Changed 10 years ago by ppurka

It's working nice. I downloaded 5.4beta1 and started the compilation with ccache enabled. Then I canceled the compilation after some time and ran make distclean and restarted all over again. Stuff that was previously compiled was flying by really fast. :)

I will have to see how much it improves when there is an upgrade.

comment:36 Changed 10 years ago by ppurka

A general question. How can I upgrade my installation with this patch in place? Trying to upgrade modifies some files in SAGE_ROOT and then the upgrade fails with the following message

Successfully installed pynac-0.2.5
Deleting temporary build directory
/home/punarbasu/Installations/sage-5.4.beta1/spkg/build/pynac-0.2.5
Making Python scripts relocatable...
Finished installing pynac-0.2.5.spkg
make[1]: Leaving directory `/home/punarbasu/Installations/sage-5.4.beta1/spkg'
make: *** [all] Error 2

real    0m49.350s
user    0m59.432s
sys     0m7.064s
Error building Sage.
Error installing Sage!
Double checking that all packages have been installed.
Downloading packages from 'http://jambu.spms.ntu.edu.sg/sage/devel/sage-5.4.rc1/spkg'.
Reading package lists...  Done!
The following packages will be upgraded:

    conway_polynomials-0.3 elliptic_curves-0.7 extcode-5.4.rc1
    gcc-4.6.3 graphs-20120404.p4 polytopes_db-20100210.p2 r-2.14.0.p6
    sage-5.4.rc1 sage_root-5.4.rc1 sage_scripts-5.4.rc1

 ** WARNING: This is a source-based upgrade, which could take hours,
 ** fail, and render your Sage install useless!!

Do you want to continue [y/N]? y
There are uncommitted changes in the Sage root repository:
M spkg/install
M spkg/standard/deps

Aborting.

comment:37 follow-ups: Changed 10 years ago by ppurka

Unfortunately, ccache doesn't help very much in the compilation of a new Sage tarball. Most of the time is taken up by ATLAS. I have a failed build of 5.4.rc4 here that failed at ATLAS, after a total compilation time of just over 3 hours.

real	187m47.272s
user	236m57.749s
sys	12m48.248s
Error building Sage.
make: *** [build] Error 1
  14218.03s user 771.96s system 133% cpu 3:07:48.73 total

EDIT: In comparison, a successful build of sage (make build) takes just over 4 hours.

Last edited 10 years ago by ppurka (previous) (diff)

comment:38 in reply to: ↑ 37 Changed 10 years ago by ohanar

I never build ATLAS, so I can't comment on how well ccache works with it, but I don't really think it matters much since everyone who would gain any sort of benefit from ccache should already be using SAGE_ATLAS_LIB. I do know that R seems to not get much out of ccache though.

comment:39 Changed 10 years ago by ppurka

I am installing a system ATLAS now. I have been bitten a bit too many times with failing ATLAS builds with sage. It's not a fault of sage; it is probably that the laptop overheats or something. It builds fine on a desktop.

That said, the root patch needs to be rebased on sage-5.4.rc. The rebased patch is available here. I forgot to upload it, but maybe you can take that, rename it, and upload it here, as the original file name.

comment:40 in reply to: ↑ 37 ; follow-up: Changed 10 years ago by robertwb

Replying to ppurka:

Unfortunately, ccache doesn't help very much in the compilation of a new Sage tarball. Most of the time is taken up by ATLAS. I have a failed build of 5.4.rc4 here that failed at ATLAS, after a total compilation time of just over 3 hours.

real	187m47.272s
user	236m57.749s
sys	12m48.248s
Error building Sage.
make: *** [build] Error 1
  14218.03s user 771.96s system 133% cpu 3:07:48.73 total

EDIT: In comparison, a successful build of sage (make build) takes just over 4 hours.

Just to check, cchache is shared across sage builds, right?

Regarding ATLAS, ideally most people will either be using a system build or match one of the pre-shipped tuning parameters.

comment:41 in reply to: ↑ 40 Changed 10 years ago by ppurka

Replying to robertwb:

Just to check, cchache is shared across sage builds, right?

Yes. It is shared across different installations.

Changed 10 years ago by ppurka

Apply to SAGE_ROOT

comment:42 Changed 10 years ago by ppurka

  • Description modified (diff)
  • Status changed from needs_review to positive_review

This has been working fine for me for several months already. Setting it to positive review.

I updated the root patch to apply to more recent sage (since sage-5.5beta onwards).

Note: See TracTickets for help on using tickets.