Opened 6 years ago

Closed 6 years ago

#18078 closed enhancement (fixed)

CMake experimental package

Reported by: isuruf Owned by:
Priority: major Milestone: sage-6.9
Component: packages: experimental Keywords:
Cc: Merged in:
Authors: Isuru Fernando Reviewers: François Bissey
Report Upstream: N/A Work issues:
Branch: cb96c05 (Commits, GitHub, GitLab) Commit: cb96c05fd4703db60f0d6762127245f981639b44
Dependencies: Stopgaps:

Status badges

Description (last modified by isuruf)

Sage experimental package for CMake

CMake version 3.2.3

Source can be found here, http://www.cmake.org/files/v3.2/cmake-3.2.3.tar.gz

Change History (50)

comment:1 Changed 6 years ago by isuruf

  • Branch set to u/isuruf/cmake_optional_package

comment:2 Changed 6 years ago by isuruf

  • Commit set to 2f172e0c13681493aaa7c8f4ab334dac55450571
  • Component changed from PLEASE CHANGE to packages: optional
  • Description modified (diff)
  • Type changed from PLEASE CHANGE to enhancement

New commits:

2f172e0Added CMake v3.2.1 optional package

comment:3 Changed 6 years ago by jdemeyer

Please word-wrap SPKG.txt to 72 characters per line.

If it's ready for review, don't forget to set this ticket to needs_review.

comment:4 Changed 6 years ago by jdemeyer

And don't forget to fill in your real name in the "Author" field.

comment:5 Changed 6 years ago by git

  • Commit changed from 2f172e0c13681493aaa7c8f4ab334dac55450571 to 2b9f83e5439bca4355cfce2e5e04cd8cfd13913c

Branch pushed to git repo; I updated commit sha1. New commits:

2b9f83eWord-wrap cmake/SPKG.txt

comment:6 Changed 6 years ago by isuruf

  • Authors set to Isuru Fernando
  • Status changed from new to needs_review

comment:7 follow-up: Changed 6 years ago by dimpase

The installation works on Linux, at least. How does one check it is not broken?

comment:8 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_work
WARNING: spkg-install is not executable, making it executable

comment:9 in reply to: ↑ 7 ; follow-up: Changed 6 years ago by jdemeyer

Replying to dimpase:

The installation works on Linux, at least. How does one check it is not broken?

cmake has a test-suite run with make test. So a spkg-check file should be added to run this.

One more remark: you don't really need to split up $MAKE and $MAKE install in two steps, just $MAKE install is sufficient.

comment:10 Changed 6 years ago by dimpase

Doesn't build on OSX 10.10.3 with the latest XCode:

 27%] Building CXX object Source/CMakeFiles/OSXScriptLauncher.dir/CPack/OSXScriptLauncher.cxx.o
[ 27%] Building CXX object Utilities/cmjsoncpp/CMakeFiles/cmjsoncpp.dir/src/lib_json/json_writer.cpp.o
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include/dispatch/dispatch.h:50:0,
                 from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFStream.h:15,
                 from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFPropertyList.h:13,
                 from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:55,
                 from /usr/local/src/sage/sage/local/var/tmp/sage/build/cmake-3.2.1/src/Source/CPack/OSXScriptLauncher.cxx:17:
/usr/local/src/sage/sage/local/include/dispatch/object.h:321:1: error: 'DISPATCH_UNAVAILABLE' does not name a type
 DISPATCH_UNAVAILABLE
 ^
/usr/local/src/sage/sage/local/include/dispatch/object.h:358:1: error: 'DISPATCH_UNAVAILABLE' does not name a type
 DISPATCH_UNAVAILABLE
 ^
/usr/local/src/sage/sage/local/include/dispatch/object.h:387:1: error: 'DISPATCH_UNAVAILABLE' does not name a type
 DISPATCH_UNAVAILABLE
 ^
/usr/local/src/sage/sage/local/include/dispatch/object.h:418:1: error: 'DISPATCH_UNAVAILABLE' does not name a type
 DISPATCH_UNAVAILABLE
 ^
make[2]: *** [Source/CMakeFiles/OSXScriptLauncher.dir/CPack/OSXScriptLauncher.cxx.o] Error 1
make[1]: *** [Source/CMakeFiles/OSXScriptLauncher.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Linking CXX static library libcmjsoncpp.a
[ 27%] Built target cmjsoncpp
make: *** [all] Error 2
Error building CMake.

real	7m15.276s
user	5m26.048s
sys	1m33.521s
************************************************************************
Error installing package cmake-3.2.1
************************************************************************
...

I suspect that we don't need the fully blown OSX cmake (we don't need OSXScriptLauncher probably), perhaps some options need to be used while building on OSX...

comment:11 Changed 6 years ago by fbissey

My experience with kitware stuff on OS X is they assume clang and most other stuff is in the too hard basket. I was helping someone get paraview on their OS X box and we had to give up using gcc starting 10.10.2. I will have a look if we can do something at booststrap time but I am not especially hopeful.

comment:12 Changed 6 years ago by fbissey

Current error is while building internal version of curl which is of course available by default on OS X. But using the system curl just push the problem further away. Attentive people will notice that the problematic header is the one we copy and modify for gcc install on 10.10.

I hate to do that, but my suggestion is for this to set CC=clang CXX=clang++ on OS X.

Last edited 6 years ago by fbissey (previous) (diff)

comment:13 Changed 6 years ago by fbissey

OK, it is worse than I thought. Not only do we get a gcc that will sometimes break, when gcc breaks so does clang because of that little header. I am wondering if we should remove that header once gcc is built. It looks like a lot of stuff that will want that header will break regardless. Thoughts?

comment:14 Changed 6 years ago by git

  • Commit changed from 2b9f83e5439bca4355cfce2e5e04cd8cfd13913c to 72a020d1f2628dd7fc7595b9cf0485a14d6fb426

Branch pushed to git repo; I updated commit sha1. New commits:

72a020dMake cmake/spkg-install executable

comment:15 in reply to: ↑ 9 Changed 6 years ago by isuruf

Replying to jdemeyer:

Replying to dimpase:

The installation works on Linux, at least. How does one check it is not broken?

cmake has a test-suite run with make test. So a spkg-check file should be added to run this.

cmake has a test suite that can be run using ctest. I wrote a spkg-check and ran the tests on Linux. 3/382 tests fail, two of which is related to curl, and the other about a sleep command. I'm not sure whether these options will be required by packages in the future. 23 - curl (Failed) 178 - CTestTestUpload (Failed) 218 - CTestTestStopTime (Failed)

One more remark: you don't really need to split up $MAKE and $MAKE install in two steps, just $MAKE install is sufficient.

Thanks. I will make this change.

comment:16 Changed 6 years ago by git

  • Commit changed from 72a020d1f2628dd7fc7595b9cf0485a14d6fb426 to 3aae9bf7905d0cb5ec63eaceef8ee9f590c87463

Branch pushed to git repo; I updated commit sha1. New commits:

3aae9bfCMake : Merge make and make install steps into one

comment:17 follow-up: Changed 6 years ago by fbissey

At the moment I would recommend people who want cmake on OS X to download it from kitware and make sure that the cmake command is in their PATH. We could put some words of that effect in spkg-install. I don't see this working on OS X anytime soon. We would need a breakthrough with gcc or clang.

comment:18 in reply to: ↑ 17 ; follow-up: Changed 6 years ago by dimpase

Replying to fbissey:

At the moment I would recommend people who want cmake on OS X to download it from kitware and make sure that the cmake command is in their PATH. We could put some words of that effect in spkg-install. I don't see this working on OS X anytime soon. We would need a breakthrough with gcc or clang.

I won't be surprised if this cmake won't be useful within Sage, as it might assume too much of the native OSX toolchain.

cmake was mentioned by me in a GSoC 2015 discussion regarding getting CSymPy into Sage; I expressed my doubts that cmake, which is used to configure CSymPy, is easy to deal with in Sage, based on some vague memories about cmake/Sage difficulties I saw. Now I recall that that was about building cmake in Sage on OSX.

comment:19 in reply to: ↑ 18 Changed 6 years ago by fbissey

Replying to dimpase:

Replying to fbissey:

At the moment I would recommend people who want cmake on OS X to download it from kitware and make sure that the cmake command is in their PATH. We could put some words of that effect in spkg-install. I don't see this working on OS X anytime soon. We would need a breakthrough with gcc or clang.

I won't be surprised if this cmake won't be useful within Sage, as it might assume too much of the native OSX toolchain.

cmake was mentioned by me in a GSoC 2015 discussion regarding getting CSymPy into Sage; I expressed my doubts that cmake, which is used to configure CSymPy, is easy to deal with in Sage, based on some vague memories about cmake/Sage difficulties I saw. Now I recall that that was about building cmake in Sage on OSX.

I don't think there is a difficulty per see in using cmake on OS X. The trouble building cmake and in fact gcc and in turn a bunch of other stuff is solely the fault of apple pushing its own stuff and not caring. gcc is currently slightly broken on OS X and unless the gcc team gets something together it will just get worse. Not being able to build cmake is just one of the many symptoms of gcc cracking while apple is pushing hard.

comment:20 Changed 6 years ago by dimpase

cmake 3.2.2 has been released, so this ticket ought to be adjusted to this version, I suppose.

comment:21 follow-up: Changed 6 years ago by jdemeyer

@fbissey: and the problem is that we really need GCC for Sage, because we need a Fortran compiler and clang doesn't (yet) support Fortran.

comment:22 in reply to: ↑ 21 ; follow-up: Changed 6 years ago by fbissey

Replying to jdemeyer:

@fbissey: and the problem is that we really need GCC for Sage, because we need a Fortran compiler and clang doesn't (yet) support Fortran.

I know. Although I have been trying very hard to think about fortran code in sage apart from lapack. I think the only fortran bits where you need a fortran compiler are in atlas, scipy and unfortunately for OS X: R.

We use the accelerate framework instead of Atlas on OS X, one less fortran thing.

A fortran compiler is needed while compiling scipy :( lots of fortran code in there.

R is more pernicious, you need a fortran compiler on OS X because there is a bug preventing us to use accelerate.

flang is just a dot on the horizon so far. At least they say they started work on it.

comment:23 Changed 6 years ago by leif

  • Milestone changed from sage-6.6 to sage-6.7

comment:24 in reply to: ↑ 22 ; follow-up: Changed 6 years ago by dimpase

Replying to fbissey:

Replying to jdemeyer:

@fbissey: and the problem is that we really need GCC for Sage, because we need a Fortran compiler and clang doesn't (yet) support Fortran.

I know. Although I have been trying very hard to think about fortran code in sage apart from lapack. I think the only fortran bits where you need a fortran compiler are in atlas, scipy and unfortunately for OS X: R.

We use the accelerate framework instead of Atlas on OS X, one less fortran thing.

A fortran compiler is needed while compiling scipy :( lots of fortran code in there.

R is more pernicious, you need a fortran compiler on OS X because there is a bug preventing us to use accelerate.

flang is just a dot on the horizon so far. At least they say they started work on it.

I don't even see dots, apart from some github repos untouched since 2013 or so.

We can consider going back to having a dedicated gfortran package, as it used to be the case before OSX 10.7 (IIRC) broken xcode trouble started, but otherwise use xcode's compilers.

comment:25 in reply to: ↑ 24 ; follow-up: Changed 6 years ago by jdemeyer

Replying to dimpase:

We can consider going back to having a dedicated gfortran package, as it used to be the case before OSX 10.7 (IIRC) broken xcode trouble started, but otherwise use xcode's compilers.

It's not just a matter of using gfortran. It's also a matter of linking code together (good luck linking a library containing pieces compiled by clang and other pieces by gfortran).

comment:26 follow-up: Changed 6 years ago by jdemeyer

About CMake: it's very ironic that the cmake authors wrote a build system which is supposed to make it easy to build cross-platform software, but they cannot make cmake itself build properly! That's really not good advertising for cmake.

comment:27 in reply to: ↑ 26 Changed 6 years ago by dimpase

Replying to jdemeyer:

About CMake: it's very ironic that the cmake authors wrote a build system which is supposed to make it easy to build cross-platform software, but they cannot make cmake itself build properly! That's really not good advertising for cmake.

they didn't use autotools, I suppose, that's why ;-)

comment:28 in reply to: ↑ 25 Changed 6 years ago by leif

Replying to jdemeyer:

Replying to dimpase:

We can consider going back to having a dedicated gfortran package, as it used to be the case before OSX 10.7 (IIRC) broken xcode trouble started, but otherwise use xcode's compilers.

It's not just a matter of using gfortran. It's also a matter of linking code together (good luck linking a library containing pieces compiled by clang and other pieces by gfortran).

And there are still parts of Sage which currently don't build with clang...

Otherwise, w.r.t. gfortran on MacOS X: Revive f2c! ;-)

comment:29 Changed 6 years ago by certik

Thanks for trying to compile cmake.

A few comments:

  • if csympy is the only package that requires cmake and cmake is impossible to build, we can also support an alternative build system in csympy
  • I really like cmake, so let me try to fix these issues first, as I would prefer to keep cmake as the build system (but it needs to reliably build on OS X, both with clang and with gcc)
  • I independently discovered lots of similar issues with building cmake with gcc on OS X 10.10 (https://github.com/hashdist/hashstack/issues/746), and just like you noticed, it builds fine with the native clang. As you can see, you can easily tell cmake not to compile in various things and rather reuse them from the system or Sage. I really believe all these issues can be fixed and I am also sure Kitware is itself interested in cmake being able to build, e.g. Robert Maynard from Kitware already commented on my issue above with some tips.

So I will have a look in the coming days and I will update this issue.

comment:30 Changed 6 years ago by leif

I guess this is still pretty much work in progress, but we certainly should use/tweak a couple of configure options for bootstrapping within Sage, such as for example

  --parallel=n            bootstrap cmake in parallel, where n is
                          number of nodes [1]

  --no-system-libs        use all cmake-provided third-party libraries
                          (default)

  --system-zlib           use system-installed zlib library
  --no-system-zlib        use cmake-provided zlib library (default)
  --system-bzip2          use system-installed bzip2 library
  --no-system-bzip2       use cmake-provided bzip2 library (default)

  ...

comment:31 Changed 6 years ago by git

  • Commit changed from 3aae9bf7905d0cb5ec63eaceef8ee9f590c87463 to c8778a93f2076dd8343e533511a7e6758b1803ab

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

e2c3532Added CMake v3.2.1 optional package
1cbb417Word-wrap cmake/SPKG.txt
79d84ccMake cmake/spkg-install executable
4a219ecCMake : Merge make and make install steps into one
2870d55CMake: use sage's bzip2 and zlib
fe8eb4aTest CMake installation
c8778a9Patch for building CMake on OSX 10.10

comment:32 Changed 6 years ago by jdemeyer

A lot has changed in the handling of optional packages recently, this should be rebased on top of Sage 6.8.beta5.

comment:33 Changed 6 years ago by git

  • Commit changed from c8778a93f2076dd8343e533511a7e6758b1803ab to 6fea35f5c78f637a50b1cf35d7fa2fb36a58af77

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

eadda32Added CMake v3.2.1 optional package
a84859eWord-wrap cmake/SPKG.txt
feadaf3Make cmake/spkg-install executable
919f12eCMake : Merge make and make install steps into one
268dee9CMake: use sage's bzip2 and zlib
def4c8aTest CMake installation
c73f9e2Patch for building CMake on OSX 10.10
6fea35fCMake : Add type as experimental

comment:34 Changed 6 years ago by fbissey

Is the OS X patch still needed with cmake 3.2.3? If not I suggest bumping to that and I will test it on my OS X box. If it builds I think we can review it positively as an experimental package. While reviewing we have pushed at the optional level of quality in review rather than experimental in my opinion.

comment:35 Changed 6 years ago by fbissey

I realise that I thought it was experimental because that's what the type says in the git commit. But this ticket says "optional" and I think we are getting there so the type will have to be changed too.

comment:36 Changed 6 years ago by git

  • Commit changed from 6fea35f5c78f637a50b1cf35d7fa2fb36a58af77 to 0096eed76cf7cc2fc99953192dc78807501e8f3a

Branch pushed to git repo; I updated commit sha1. New commits:

0096eedCMake : updated version to 3.2.3

comment:37 Changed 6 years ago by isuruf

  • Component changed from packages: optional to packages: experimental
  • Description modified (diff)
  • Status changed from needs_work to needs_review
  • Summary changed from CMake optional package to CMake experimental package

Yes, the patch is still needed for 3.2.3

I added experimental because even with the patch 6 tests out of 387 tests in OSX 10.10 fail. Once it is ready to become optional, I'll update the type to optional.

comment:38 Changed 6 years ago by jdemeyer

  • Status changed from needs_review to needs_work
Invalid checksum for cached file /usr/local/src/sage-config/upstream/cmake-3.2.3.tar.gz, deleting

comment:39 Changed 6 years ago by git

  • Commit changed from 0096eed76cf7cc2fc99953192dc78807501e8f3a to b65bb43a99a3290e10ccf6f752eb160aafb95e89

Branch pushed to git repo; I updated commit sha1. New commits:

b65bb43CMake : update checksums

comment:40 Changed 6 years ago by isuruf

Sorry about that, chrome seems to be playing tricks on me. It's updated now.

comment:41 Changed 6 years ago by isuruf

It seems that the curl library included with CMake is outdated and needs to be built separately for CMake to work in OSX.

comment:42 Changed 6 years ago by dimpase

  • Milestone changed from sage-6.7 to sage-6.8

Does it really need curl? OSX does ship curl just fine...

comment:43 Changed 6 years ago by isuruf

There are some features that involve downloading. That's why curl is used.

We can use system installed curl for OSX and build curl for other systems.

comment:44 Changed 6 years ago by dimpase

curl is really very standard thing on Linux, installed by default. I don't know why cmake needs it for anything but Windows...

comment:45 Changed 6 years ago by git

  • Commit changed from b65bb43a99a3290e10ccf6f752eb160aafb95e89 to d71e284bcb8127523ffff4a0229fa152c0064976

Branch pushed to git repo; I updated commit sha1. New commits:

d71e284CMake : Use system curl

comment:46 Changed 6 years ago by isuruf

I pushed in a commit to use system curl when building cmake.

@dimpase, yes, there's really no need to build curl as it is installed in OSX and Linux. Building cmake in windows is not supported, so no use there either.

comment:47 Changed 6 years ago by git

  • Commit changed from d71e284bcb8127523ffff4a0229fa152c0064976 to cb96c05fd4703db60f0d6762127245f981639b44

Branch pushed to git repo; I updated commit sha1. New commits:

cb96c05Use system-curl only in OSX and build curl in Linux

comment:48 Changed 6 years ago by isuruf

  • Status changed from needs_work to needs_review

comment:49 Changed 6 years ago by fbissey

  • Milestone changed from sage-6.8 to sage-6.9
  • Reviewers set to François Bissey
  • Status changed from needs_review to positive_review

Let's get this in, this is experimental after all.

comment:50 Changed 6 years ago by vbraun

  • Branch changed from u/isuruf/cmake_optional_package to cb96c05fd4703db60f0d6762127245f981639b44
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.