#9187 closed enhancement (fixed)
Update ECL's spkg-install for building multiple spkgs in parallel
Reported by: | mpatel | Owned by: | drkirkby |
---|---|---|---|
Priority: | minor | Milestone: | sage-4.5 |
Component: | packages: standard | Keywords: | |
Cc: | drkirkby, jhpalmieri, leif, jsp | Merged in: | sage-4.5.rc1 |
Authors: | Mitesh Patel | Reviewers: | John Palmieri, John Cremona |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description
To build ECL with SAGE_PARALLEL_SPKG_BUILD="yes"
on Mac OS X, we need to add, e.g.,
MAKEFLAGS= export MAKEFLAGS
to the package's spkg-install
.
Please see #8306 about building spkgs in parallel. For MAKEFLAGS
, see the GNU Make manual.
Attachments (2)
Change History (34)
comment:1 Changed 12 years ago by
comment:2 Changed 12 years ago by
- Cc drkirkby jhpalmieri leif added
- Status changed from new to needs_review
I've put a new spkg at
(Or should it be p1?) This is based against the package mentioned in this comment at #8951.
comment:3 Changed 12 years ago by
- Cc jsp added
- Owner changed from tbd to drkirkby
Unfortunately, there are two spkgs on #8951 and neither have the patches from #8089 and both claim in SPKG.txt to have things they do not.
I intend creating a 10.4.1 package which has this, and all other relevant fixes in the one package. Since no ecl-10.4.1.spkg has been merged, this should be called ecl-10.4.1.spkg. The patch levels start at 0, not 1, but in this case as nothing has been merged, there is no need to increment the patch level.
Please see #9264, which will have these changes, those which are listed as being in #8591 but are not, and also that at #8089
Dave
comment:4 Changed 12 years ago by
- Status changed from needs_review to positive_review
The attached patch is ok and deserves positive review. However, the .spkg attached is not ideal, as it overlooks some other changes which need merging. I think a new .spkg needs to be created, which has all the minor changes together.
comment:5 Changed 12 years ago by
comment:6 Changed 12 years ago by
- Resolution set to duplicate
- Status changed from positive_review to closed
comment:7 Changed 12 years ago by
- Milestone changed from sage-5.0 to sage-duplicate/invalid/wontfix
comment:8 Changed 12 years ago by
- Milestone changed from sage-duplicate/invalid/wontfix to sage-4.5.1
- Resolution duplicate deleted
- Status changed from closed to new
comment:9 follow-up: ↓ 11 Changed 12 years ago by
Here's a new spkg based on #9474:
http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg
It works for me with 4.5.rc0 + #7379 on sage.math, but I have not yet had a chance to test it on bsd.math.
comment:10 Changed 12 years ago by
This works for me on sage.math and on an OS X box running 10.6, in both cases building spkg's in parallel. I'm building on t2 now; if there are problems, I'll report them.
comment:11 in reply to: ↑ 9 ; follow-up: ↓ 12 Changed 12 years ago by
Replying to mpatel:
Here's a new spkg based on #9474:
http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg
It works for me with 4.5.rc0 + #7379 on sage.math, but I have not yet had a chance to test it on bsd.math.
It works for me on bsd.math - at least ecl builds ok. I've not at this time completed the build, but ecl has built ok.
comment:12 in reply to: ↑ 11 Changed 12 years ago by
Replying to drkirkby:
It works for me on bsd.math - at least ecl builds ok. I've not at this time completed the build, but ecl has built ok.
That's what I meant, too. The same is also true on t2 now: this version of ecl has built successfully as part of a parallel build, but the full build has several more hours to go...
comment:13 follow-up: ↓ 14 Changed 12 years ago by
I am testing a new build (on 64-bit ubuntu) of 4.5.rc0 + http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg with SAGE_PARALLEL_SPKG_BUILD="yes" (which I have never tried before) and will report back. Though these ecl tickets are so confusing that I am not sure this will tell anyone anything useful, I thought it would be fun. (The other users of the machine may not agree, of course!)
comment:14 in reply to: ↑ 13 ; follow-up: ↓ 15 Changed 12 years ago by
Replying to cremona:
I am testing a new build (on 64-bit ubuntu) of 4.5.rc0 + http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg with SAGE_PARALLEL_SPKG_BUILD="yes" (which I have never tried before) and will report back. Though these ecl tickets are so confusing that I am not sure this will tell anyone anything useful, I thought it would be fun. (The other users of the machine may not agree, of course!)
I think you test results will be very useful, as there is a distinct lack of direct evidence for this working on Linux. There is evidence it is working on OS X and Solaris.
You also need to export MAKE, with something like
$ export MAKE="make -j6"
or however many threads you want. Do not put a space between the 'j' and the '6'.
Dave
comment:15 in reply to: ↑ 14 Changed 12 years ago by
Replying to drkirkby:
Replying to cremona:
I am testing a new build (on 64-bit ubuntu) of 4.5.rc0 + http://sage.math.washington.edu/home/mpatel/trac/9187/ecl-10.2.1.p1.spkg with SAGE_PARALLEL_SPKG_BUILD="yes" (which I have never tried before) and will report back. Though these ecl tickets are so confusing that I am not sure this will tell anyone anything useful, I thought it would be fun. (The other users of the machine may not agree, of course!)
I think you test results will be very useful, as there is a distinct lack of direct evidence for this working on Linux. There is evidence it is working on OS X and Solaris.
You also need to export MAKE, with something like
$ export MAKE="make -j6"
or however many threads you want. Do not put a space between the 'j' and the '6'.
Thanks -- I use -j 10, and luckily the machine is in an air-conditioned room!
Dave
comment:16 follow-up: ↓ 17 Changed 12 years ago by
John,
I'm not sure if you mis-understood me there.
$ export MAKE="make -j 10"
will not work as well as it could do. The .spkg's will build in parallel, but when it gets to building the Sage library, that will only build serially. You should use
$ export MAKE="make -j10"
I assume you are aware you also need a library patch
http://trac.sagemath.org/sage_trac/raw-attachment/ticket/7379/trac-7379-decorator-defaults.patch
(only the library patch, not the sagenb patch on that ticket). Only part of #7379 got merged into 4.5.rc0
Dave
comment:17 in reply to: ↑ 16 Changed 12 years ago by
Replying to drkirkby:
John,
I'm not sure if you mis-understood me there.
$ export MAKE="make -j 10"
will not work as well as it could do. The .spkg's will build in parallel, but when it gets to building the Sage library, that will only build serially. You should use
$ export MAKE="make -j10"
It's ok, I knew that...
I assume you are aware you also need a library patch
http://trac.sagemath.org/sage_trac/raw-attachment/ticket/7379/trac-7379-decorator-defaults.patch
... and that. 83 spkgs now installed and counting....
(only the library patch, not the sagenb patch on that ticket). Only part of #7379 got merged into 4.5.rc0
Dave
comment:18 follow-up: ↓ 20 Changed 12 years ago by
- Reviewers set to John Palmieri, John Cremona
- Status changed from new to needs_review
I built this successfully on sage.math earlier. I'm willing to give it a positive review right now. John (Cremona): I'll give you the last word. If everything works for you, could you give this a positive review?
comment:19 Changed 12 years ago by
I think this ecl-10.2.1.p1.spkg can be merged into rc1, but could you MacOS X testers check which version of Boehm GC is used during install (i.e., the one shipped with Sage or the one included in ECL)?
$ grep -i boehm spkg/logs/ecl-10.2.1.p1.log | head -n 2
should show soemthing like
checking whether we can use the existing Boehm-Weiser library ... yes checking if we need to copy GC private headers ... checking if we use Boehm-Demers-Weiser precise garbage collector... yes
or
checking if we use Boehm-Demers-Weiser precise garbage collector... yes configure: Configuring included Boehm GC library:
(The latter is from Dave's install log on bsd.math/MacOS X 10.4.0.)
comment:20 in reply to: ↑ 18 ; follow-up: ↓ 26 Changed 12 years ago by
Replying to jhpalmieri:
I built this successfully on sage.math earlier. I'm willing to give it a positive review right now. John (Cremona): I'll give you the last word. If everything works for you, could you give this a positive review?
It finished building, now building docs. Then I'll do a ptestlong and report back -- currently I don't know if what was built actually works!
comment:21 Changed 12 years ago by
On my OS X box, it says:
checking if we use Boehm-Demers-Weiser precise garbage collector... yes ... configure: Configuring included Boehm GC library:
I'm not an expert in these things, but in src/src/configure, it says:
darwin*) thehost='darwin' shared='yes' ... # The Boehm-Weiser GC library shipped with Fink does not work # well with our signal handler. enable_boehm=included
which leads me to believe that on OS X, it will be a little work to use Sage's installation of boehm_gc. I think this should go on a follow-up ticket, and it should probably be done in coordination with the ecl people.
comment:22 follow-up: ↓ 23 Changed 12 years ago by
(That is, on OS X, it looks like any setting to the --enable-boehm
flag gets overridden.)
comment:23 in reply to: ↑ 22 ; follow-up: ↓ 24 Changed 12 years ago by
Replying to jhpalmieri:
(That is, on OS X, it looks like any setting to the
--enable-boehm
flag gets overridden.)
That's odd, since Sage's Boehm GC includes a patch for MacOS X 10.6.
I agree we should open a new ticket for that and perhaps include a p2 spkg in 4.5.rc2(?), 4.5.final or 4.5.1.
The package has to be cleaned up anyway, and perhaps we switch back to ECL 10.4.1.
comment:24 in reply to: ↑ 23 ; follow-up: ↓ 25 Changed 12 years ago by
Replying to leif:
Replying to jhpalmieri: I agree we should open a new ticket for that and perhaps include a p2 spkg in 4.5.rc2(?), 4.5.final or 4.5.1.
4.5.1.
comment:25 in reply to: ↑ 24 Changed 12 years ago by
comment:26 in reply to: ↑ 20 Changed 12 years ago by
- Status changed from needs_review to positive_review
Replying to cremona:
Replying to jhpalmieri:
I built this successfully on sage.math earlier. I'm willing to give it a positive review right now. John (Cremona): I'll give you the last word. If everything works for you, could you give this a positive review?
It finished building, now building docs. Then I'll do a ptestlong and report back -- currently I don't know if what was built actually works!
All tests passed == +++
comment:27 Changed 12 years ago by
- Merged in set to sage-4.5.rc1
- Milestone changed from sage-4.5.1 to sage-4.5
- Resolution set to fixed
- Status changed from positive_review to closed
comment:28 Changed 12 years ago by
Replying to leif:
Replying to jhpalmieri:
(That is, on OS X, it looks like any setting to the
--enable-boehm
flag gets overridden.)That's odd, since Sage's Boehm GC includes a patch for MacOS X 10.6.
Just for the record: The patch just avoids a dumb deprecation error(!), and it is already included in ECL's Boehm GC (otherwise ECL wouldn't have built on MacOS X 10.6).
comment:29 follow-ups: ↓ 30 ↓ 31 Changed 12 years ago by
For a 4.5 final, we should remove the unnecessary baggage, though (as SPKG.txt
instructs, btw):
leif@portland:~/Sage/spkgs/ecl-10.2.1.p1$ du -ch src/msvc/ src/contrib/encodings/ 16K src/msvc/ecl 8.0K src/msvc/gc 8.0K src/msvc/gmp/mpz 32K src/msvc/gmp/mpn/generic 76K src/msvc/gmp/mpn/amd64i 48K src/msvc/gmp/mpn/x86i/pentium4/sse2 32K src/msvc/gmp/mpn/x86i/pentium4/mmx 92K src/msvc/gmp/mpn/x86i/pentium4 48K src/msvc/gmp/mpn/x86i/p6/mmx 8.0K src/msvc/gmp/mpn/x86i/p6/p3mmx 128K src/msvc/gmp/mpn/x86i/p6 300K src/msvc/gmp/mpn/x86i 412K src/msvc/gmp/mpn 12K src/msvc/gmp/build.vc8/gen-fib 16K src/msvc/gmp/build.vc8/gen-fac_ui 28K src/msvc/gmp/build.vc8/lib_gmpxx 256K src/msvc/gmp/build.vc8/dll_gmp_amd64 16K src/msvc/gmp/build.vc8/gen-bases 372K src/msvc/gmp/build.vc8/lib_gmp 240K src/msvc/gmp/build.vc8/dll_gmp_p4 40K src/msvc/gmp/build.vc8/lib_gmp_p0 36K src/msvc/gmp/build.vc8/lib_gmp_p3 240K src/msvc/gmp/build.vc8/dll_gmp_p0 180K src/msvc/gmp/build.vc8/dll_mpfr 28K src/msvc/gmp/build.vc8/gen-psqr 72K src/msvc/gmp/build.vc8/lib_gmp_gc 40K src/msvc/gmp/build.vc8/lib_gmp_p4 48K src/msvc/gmp/build.vc8/lib_gmp_amd64 436K src/msvc/gmp/build.vc8/dll_gmp_gc 240K src/msvc/gmp/build.vc8/dll_gmp_p3 208K src/msvc/gmp/build.vc8/lib_mpfr 4.8M src/msvc/gmp/build.vc8 5.2M src/msvc/gmp 8.0K src/msvc/c 16K src/msvc/util 12K src/msvc/doc 5.3M src/msvc/ 544K src/contrib/encodings/ 5.8M total leif@portland:~/Sage/spkgs/ecl-10.2.1.p1$ tar cj src/msvc/ src/contrib/encodings/ | wc -c 859300
comment:30 in reply to: ↑ 29 Changed 12 years ago by
Replying to leif:
For a 4.5 final, we should remove the unnecessary baggage, though (as
SPKG.txt
instructs, btw):
That's up to Robert, but I tend to feel saving about 840 KB is of relatively low importance, and can be left for 4.5.1. Especially considering there is a new version of ECL available, and we would expect to resolve the ECL/Maxima issues some time in the near future. So the ecl package will probably be updated for 4.5.1.
Dave
comment:31 in reply to: ↑ 29 ; follow-up: ↓ 32 Changed 12 years ago by
Replying to leif:
For a 4.5 final, we should remove the unnecessary baggage
Please open a ticket and make a p2 with only these changes. I think it's too late for 4.5.
comment:32 in reply to: ↑ 31 Changed 12 years ago by
Replying to rlm:
Replying to leif:
For a 4.5 final, we should remove the unnecessary baggage
Please open a ticket and make a p2 with only these changes. I think it's too late for 4.5.
Ticket up at #9493.
I unfortunately cannot upload an spkg because I don't have an account on sage.math.
(Perhaps I'll ask again someone to upload it from an e-mail attachment, but it's still fairly large.)
I think we should work from #8951.