Opened 4 years ago

Closed 4 years ago

Last modified 4 years ago

#15433 closed task (fixed)

Port to OSX 10.9

Reported by: ohanar Owned by:
Priority: blocker Milestone: sage-5.13
Component: porting Keywords:
Cc: yomcat, jhpalmieri Merged in: sage-5.13.rc0
Authors: R. Andrew Ohana Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: u/ohanar/10.9-port Commit:
Dependencies: Stopgaps:

Description (last modified by jdemeyer)

Sage fails to build on OSX 10.9.

Issues:

  1. scipy doesn't play nicely with the new accelerate framework headers
  2. R depends on CoreData? which uses non-standard sqlite3 modules (and this affects us since we set *_LIBRARY_PATH and install a custom copy of sqlite3)
  3. g++ and 10.9's cdefs.h header are currently incompatible, leading to a broken libstc++
  4. accelerate framework no longer pretends to be atlas for linking purposes

To Apply:

  1. Apply trac15433.patch to the Sage Library repository
  2. gcc-4.7.3.p1.spkg
  3. sqlite-3.7.17.p1.spkg
  4. scipy-0.12.0.p1.spkg

Attachments (4)

trac15433.patch (647 bytes) - added by ohanar 4 years ago.
gcc-4.7.3.p1.diff (1.2 KB) - added by ohanar 4 years ago.
for review purposes
scipy-0.12.0.p1.diff (949 bytes) - added by ohanar 4 years ago.
for review purposes
sqlite-3.7.17.p1.diff (1.5 KB) - added by jdemeyer 4 years ago.

Download all attachments as: .zip

Change History (57)

comment:1 Changed 4 years ago by ohanar

The attached branch should fix everything except for the polybori issue. This should be checked on older OSX versions, since I've made some changes that are not 10.9 specific (I've checked these changes against bsd.math, and they work fine there).

As for the polybori issue, I'm not experiencing it anymore and I don't know what fixed it for me. I was going to setup a 10.9 VM, but unfortunately Apple refuses to let me download 10.9 on the machine I'm currently using -- however, I should be able to get a copy sometime in the few days.

I will update spkgs/add mercurial patches if we can fix this in time for 5.13, but for now it is much easier for me to work off of the git version.

comment:2 Changed 4 years ago by yomcat

  • Cc yomcat added

comment:3 Changed 4 years ago by vbraun

I'm getting the following:

$ make
cd build && \
"../build/pipestatus" \
	"env SAGE_PARALLEL_SPKG_BUILD='' ./install all 2>&1" \
	"tee -a ../logs/install.log"
./install: line 503: syntax error near unexpected token `then'
./install: line 503: `if { uname -sr | grep 'Darwin' } &>/dev/null; then'
make: *** [build] Error 2

$ uname -sr
Linux 3.11.7-200.fc19.x86_64

comment:4 Changed 4 years ago by jhpalmieri

  • Cc jhpalmieri added

comment:5 Changed 4 years ago by git

  • Commit changed from 7a0879bde6a2e557b7a4056eba398eeaef60f54f to 3ad56a494d4f8085189695546d4ff7480155608a

Branch pushed to git repo; I updated commit sha1. New commits:

3ad56a4build/install: add missing ;

comment:6 follow-up: Changed 4 years ago by jhpalmieri

Here's a stupid git question: what command do I give to check out this branch? I mean, if I'm on an OS X Mavericks system and I don't have a working copy of Sage with the git devel tools, I can't just do ./sage -dev …, but there must be some git command using the branch name or something. So what is it?

comment:7 Changed 4 years ago by jhpalmieri

My guess about polybori, by the way, is that it was fixed by the changes to gcc.

comment:8 in reply to: ↑ 6 Changed 4 years ago by ohanar

Replying to jhpalmieri:

Here's a stupid git question: what command do I give to check out this branch?

Assuming you have git installed and in your path, you can do

$ git clone git://trac.sagemath.org/sage.git -b u/ohanar/10.9-port

to make a clone of the git repository, or if you already have a clone, you can do

$ git fetch git://trac.sagemath.org/sage.git u/ohanar/10.9-port
$ git checkout -b u/ohanar/10.9-port FETCH_HEAD

from within the repository.

comment:9 follow-up: Changed 4 years ago by zabrocki

How do I resolve this problem?

sh: line 1: 59700 Trace/BPT trap: 5       /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk / -find ar 2> /dev/null
ar: error: unable to find utility "ar", not a developer tool or in PATH
make[3]: *** [libsymmetrica.a] Error 72
make[3]: Target `all' not remade because of errors.
Error building Symmetrica.

because I do see

-bash:/Applications/sage-5.13.beta1 $ /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk / -find ar
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar
-bash:/Applications/sage-5.13.beta1 $ echo $PATH
.:/usr/bin:/bin:/usr/sbin

comment:10 in reply to: ↑ 9 Changed 4 years ago by ohanar

Replying to zabrocki:

How do I resolve this problem?

Looks like more or less the same problem as polybori. That said, symmetrica was updated in 5.13.beta2, so you may want to see if you still experience the issue with a more recent beta.

comment:11 Changed 4 years ago by jhpalmieri

I don't know how the new build system works, but if you modify a file like build/pkgs/gcc/spkg-install, for example, should you also modify build/pkgs/gcc/package-version.txt so that the new version gets built? I have a version of Sage/git from early October, and when I cloned this branch and ran make, gcc was not rebuilt — it didn't even try, the log file was not modified — while scipy, for example, just returned the message Package scipy-0.12.0.p0 is already installed.

Building the whole thing from scratch worked well, though, and all tests passed.

Regarding the patch to gcc/spkg-install, for consistency's sake among other things, would it be a good idea to check the OS X version number the same way it's done in prereqs? That is, DARWIN_VERSION=`uname -r | cut '-d.' -f1`?

comment:12 Changed 4 years ago by zabrocki

The directory name is sage-5.13.beta1 is probably misleading since it is the name of the 'git' copy of sage. I did a checkout of the branch and it stops at the end of the compile of symmetrica with a make -k. If I just run make I don't get as far.

comment:13 Changed 4 years ago by ohanar

I agree that gcc should have a patch version bump, but I don't think that scipy should: scipy issue is purely a build-time issue, unlike gcc's.

For gcc/spkg-install I just tried to be consistent with what was already there.

comment:14 Changed 4 years ago by git

  • Commit changed from 3ad56a494d4f8085189695546d4ff7480155608a to 119372bfd995f1d06aaf11e774456a8abb176cfe

Branch pushed to git repo; I updated commit sha1. New commits:

119372bgcc: version bump to force rebuild on OSX 10.9

comment:15 Changed 4 years ago by ohanar

  • Authors set to R. Andrew Ohana
  • Status changed from new to needs_review

jhpalmieri: the change to gcc resolved the polybori and symmetrica issues on my second machine as well

zabrocki: try rebuilding with the latest commit

I'm marking this as needs review, although if someone could test on OSX 10.4, that would be great.

comment:16 Changed 4 years ago by yomcat

No issues at all building on 10.9 for me -- all test passed.

comment:17 Changed 4 years ago by zabrocki

Andrew, you saved me again! Brilliant! It worked for me too. Successful build and all tests pass. I had only to make all because files compiled pre-fix were messing things up (or instead, I just untar'ed another copy and compiled that version).

comment:18 Changed 4 years ago by git

  • Commit changed from 119372bfd995f1d06aaf11e774456a8abb176cfe to 9fb89a2e18a84261b3bb61a0bd9dcc6f3cf25b18

Branch pushed to git repo; I updated commit sha1. New commits:

9fb89a2build/install: add comment explaining why we don't build sqlite on OSX

comment:19 follow-up: Changed 4 years ago by jpflori

Random remark: I would have put the logic for installing or not sqlite3 in the spkg-install script of sqlite3, just as we do for cephes for example.

comment:20 in reply to: ↑ 19 Changed 4 years ago by jhpalmieri

Replying to jpflori:

Random remark: I would have put the logic for installing or not sqlite3 in the spkg-install script of sqlite3, just as we do for cephes for example.

Atlas also. That makes sense to me.

comment:21 Changed 4 years ago by git

  • Commit changed from 9fb89a2e18a84261b3bb61a0bd9dcc6f3cf25b18 to cfc31f333b25626941353545126dbc4c6dbb01f1

Branch pushed to git repo; I updated commit sha1. New commits:

cfc31f3sqlite: move platform disabling to spkg-install like atlas and cephes

comment:22 Changed 4 years ago by jhpalmieri

Now you also need to change the instructions in src/bin/sage about running sage -i sqlite3: that will have no effect on OS X. Note also that sqlite3 appears twice, once in usage and once in usage_advanced. Maybe

  • src/bin/sage

    diff --git a/src/bin/sage b/src/bin/sage
    index 7eca9b0..df51f76 100755
    a b usage() { 
    2525    echo "  -python [...]       -- run the Python interpreter"
    2626    echo "  -R [...]            -- run Sage's R with given arguments"
    2727    echo "  -singular [...]     -- run Sage's singular with given arguments"
     28    test -x "$SAGE_LOCAL/bin/sqlite3" || \
    2829    echo "  -sqlite3 [...]      -- run Sage's sqlite3 with given arguments"
    2930    echo "  -root               -- print the Sage root directory"
    3031    echo "  --nodotsage         -- run Sage without using the user's .sage directory:"
    usage_advanced() { 
    102103    echo "  -scons [...]        -- run Sage's scons"
    103104    echo "  -sh [...]           -- run \$SHELL ($SHELL) with Sage environment variables"
    104105    echo "  -singular [...]     -- run Sage's singular with given arguments"
    105     echo "  -sqlite3 [...]      -- run Sage's sqlite3 with given arguments"
    106106    test -x "$SAGE_LOCAL/bin/sqlite3" || \
    107     echo "                         (not installed currently, run sage -i sqlite3)"
     107    echo "  -sqlite3 [...]      -- run Sage's sqlite3 with given arguments"
    108108    echo "  -twistd [...]       -- run Twisted server"
    109109
    110110    echo

Or you could do

  • src/bin/sage

    diff --git a/src/bin/sage b/src/bin/sage
    index 7eca9b0..5be445b 100755
    a b usage() { 
    2626    echo "  -R [...]            -- run Sage's R with given arguments"
    2727    echo "  -singular [...]     -- run Sage's singular with given arguments"
    2828    echo "  -sqlite3 [...]      -- run Sage's sqlite3 with given arguments"
     29    test -x "$SAGE_LOCAL/bin/sqlite3" || \
     30    echo "                         (NOTE: not installed on OS X)"
    2931    echo "  -root               -- print the Sage root directory"
    3032    echo "  --nodotsage         -- run Sage without using the user's .sage directory:"
    3133    echo "                         create and use a temporary .sage directory instead"
    usage_advanced() { 
    104106    echo "  -singular [...]     -- run Sage's singular with given arguments"
    105107    echo "  -sqlite3 [...]      -- run Sage's sqlite3 with given arguments"
    106108    test -x "$SAGE_LOCAL/bin/sqlite3" || \
    107     echo "                         (not installed currently, run sage -i sqlite3)"
     109    echo "                         (NOTE: not installed on OS X)"
    108110    echo "  -twistd [...]       -- run Twisted server"
    109111
    110112    echo

comment:23 Changed 4 years ago by git

  • Commit changed from cfc31f333b25626941353545126dbc4c6dbb01f1 to 3f36a9e459741b609d176251285f969485c03478

Branch pushed to git repo; I updated commit sha1. New commits:

3f36a9eremove --sqlite3 option if sqlite is not installed

comment:24 follow-up: Changed 4 years ago by jhpalmieri

Would it be a good idea on OS X to create a symlink at local/bin/sqlite3 pointing to /usr/bin/sqlite3 (or whatever the output of command -v sqlite3 is)? Then sage --sqlite3 would still work.

comment:25 in reply to: ↑ 24 Changed 4 years ago by was

Replying to jhpalmieri:

Would it be a good idea on OS X to create a symlink at local/bin/sqlite3 pointing to /usr/bin/sqlite3 (or whatever the output of command -v sqlite3 is)? Then sage --sqlite3 would still work.

See trac #12627 which would fix this.

comment:26 follow-up: Changed 4 years ago by jhpalmieri

As I noted at #12627, that ticket is somewhat controversial, and I'm not sure it's a good idea to get it merged soon. Something much more targeted, like making a symlink for sqlite3 only on OS X, seems more palatable.

comment:27 Changed 4 years ago by was

REMARKS:

  • In build/pkgs/gcc/spkg-install and build/pkgs/sqlite/spkg-install ¶, change "OSX 10.9" to "OS X 10.9". Also "posed" --> "posted".
  • If trac #12627 gets in (which it should if rebased), then this would be deleted:
     if test -x "$SAGE_LOCAL/bin/sqlite3" && [ "$1" = '-sqlite3' -o "$1" = '--sqlite3' ]; then 
    

I am for having #12627 merged, since I can easily imagine users, scripts, workflows, whatever that have "sage --sqlite3" in them, and would be broken by the above change.

Modulo testing (on 10.4, 10.5, 10.6, 10.7, 10.8, 10.9), and the above comment regarding #12627, I think this patch looks excellent.

comment:28 in reply to: ↑ 26 Changed 4 years ago by was

Replying to jhpalmieri:

As I noted at #12627, that ticket is somewhat controversial, and I'm not sure it's a good idea to get it merged soon. Something much more targeted, like making a symlink for sqlite3 only on OS X, seems more palatable.

OK, good point -- symlinking temporarily seems reasonable, since fully supporting OS X 10.9 asap is important.

comment:29 Changed 4 years ago by git

  • Commit changed from 3f36a9e459741b609d176251285f969485c03478 to 2766ab35a1e88af4f36e46b2eb29eb9893707cb2

Branch pushed to git repo; I updated commit sha1. New commits:

2766ab3gcc: fix comment typos in install script
a64c2d7symlink system sqlite3 on OS X

comment:30 Changed 4 years ago by git

  • Commit changed from 2766ab35a1e88af4f36e46b2eb29eb9893707cb2 to 7547fceeb85afc0b6b932cb2487e61fb94fef405

Branch pushed to git repo; I updated commit sha1. New commits:

7547fceremove a couple of now unused lines

Changed 4 years ago by ohanar

comment:31 Changed 4 years ago by ohanar

  • Description modified (diff)

Ok, since this seems to be all but positively reviewed now, I've backported this to mercurial land.

comment:32 follow-up: Changed 4 years ago by jpflori

I've had a quick look at the git branch and it seems fine. Could you please attach the spkg diffs to ease review?

Changed 4 years ago by ohanar

for review purposes

Changed 4 years ago by ohanar

for review purposes

comment:33 in reply to: ↑ 32 Changed 4 years ago by ohanar

Replying to jpflori:

I've had a quick look at the git branch and it seems fine. Could you please attach the spkg diffs to ease review?

Done.

comment:34 follow-up: Changed 4 years ago by vbraun

  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

I've tested it on the buildbot and it doesn't impact other arches. Presumably it does the trick on OSX 10.9, so we should ship it asap.

comment:35 in reply to: ↑ 34 Changed 4 years ago by yomcat

Replying to vbraun:

Presumably it does the trick on OSX 10.9, so we should ship it asap.

I can confirm that it does indeed do the trick on Mavericks.

comment:36 Changed 4 years ago by jdemeyer

  • Status changed from positive_review to needs_work

When upgrading from older versions of Sage on OS X:

sage -t --long devel/sage/sage/tests/cmdline.py
**********************************************************************
File "devel/sage/sage/tests/cmdline.py", line 595, in sage.tests.cmdline.test_executable
Failed example:
    out.startswith("3.")
Expected:
    True
Got:
    False
**********************************************************************
File "devel/sage/sage/tests/cmdline.py", line 597, in sage.tests.cmdline.test_executable
Failed example:
    err
Expected:
    ''
Got:
    'dyld: Library not loaded: /Users/buildbot/build/sage/bsd-1/bsd_64_upgrade/sage-4.6.2/local/lib/libreadline.6.1.dylib\n  Referenced from: /Users/buildbot/build/sage/bsd-1/bsd_upgrade_4.6.2/build/sage-5.13.rc0/local/bin/sqlite3\n  Reason: image not found\n'
**********************************************************************
File "devel/sage/sage/tests/cmdline.py", line 599, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    -5
**********************************************************************

comment:37 follow-up: Changed 4 years ago by jdemeyer

Does the sqlite3 issue appear also on OS X < 10.9 systems? If not, then can we keep building sqlite3 on those systems and only skip it for OS X 10.9?

comment:38 Changed 4 years ago by jdemeyer

  • Priority changed from major to blocker

comment:39 in reply to: ↑ 37 Changed 4 years ago by ohanar

Replying to jdemeyer:

Does the sqlite3 issue appear also on OS X < 10.9 systems? If not, then can we keep building sqlite3 on those systems and only skip it for OS X 10.9?

I don't think so, although only doing it for >= 10.9 seems dangerous as I could easily see apple changing one of their libraries that r uses in a minor update.

comment:40 Changed 4 years ago by jdemeyer

Or we could check whether sqlite already exists and is not a symbolic link:

if [ -x "$SAGE_LOCAL/bin/sqlite3" -a ! -h "$SAGE_LOCAL/bin/sqlite3" ]; then
    # Install sqlite3 anyway
fi

comment:41 Changed 4 years ago by ohanar

  • Status changed from needs_work to needs_review

ok, updated the sqlite spkg, it should now gracefully handle upgrading.

comment:42 Changed 4 years ago by jdemeyer

Looks good on first sight, I will test it on the buildbots and set positive_review if that works.

comment:43 Changed 4 years ago by jdemeyer

  • Status changed from needs_review to needs_work

On OS X 10.4, this causes essentially every doctest involving databases to fail, for example

sage -t --long devel/sage/doc/en/tutorial/tour_advanced.rst
**********************************************************************
File "devel/sage/doc/en/tutorial/tour_advanced.rst", line 251, in doc.en.tutorial.tour_advanced
Failed example:
    db.curves(37)
Exception raised:
    Traceback (most recent call last):
      File "/Users/buildbot/build/sage/moufang-1/moufang_full/build/sage-5.13.rc0/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 479, in _run
        self.execute(example, compiled, test.globs)
      File "/Users/buildbot/build/sage/moufang-1/moufang_full/build/sage-5.13.rc0/local/lib/python2.7/site-packages/sage/doctest/forker.py", line 838, in execute
        exec compiled in globs
      File "<doctest doc.en.tutorial.tour_advanced[1]>", line 1, in <module>
        db.curves(Integer(37))
      File "/Users/buildbot/build/sage/moufang-1/moufang_full/build/sage-5.13.rc0/local/lib/python2.7/site-packages/sage/databases/cremona.py", line 769, in curves
        + 'curve=class||1 AND conductor=?',(int(N),)):
    OperationalError: ambiguous column name: class
**********************************************************************

comment:44 Changed 4 years ago by jhpalmieri

So maybe we should only use the system sqlite3 on OS X 10.9 and above?

comment:45 Changed 4 years ago by ohanar

  • Status changed from needs_work to needs_review

Ok, I've modified the spkg so it only uses the system sqlite on 10.9 and above

comment:46 Changed 4 years ago by jdemeyer

  • Status changed from needs_review to needs_work

You probably didn't mean to remove

# Old OS X systems need -DSQLITE_WITHOUT_ZONEMALLOC
if uname -sr |grep 'Darwin [0-8][.]' >/dev/null; then
    export CPPFLAGS="$CPPFLAGS -DSQLITE_WITHOUT_ZONEMALLOC"
fi

comment:47 Changed 4 years ago by jdemeyer

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Changed 4 years ago by jdemeyer

comment:48 Changed 4 years ago by ohanar

  • Status changed from needs_review to positive_review

Sorry, busy week. Yes, this looks fine to me.

comment:49 Changed 4 years ago by jdemeyer

  • Merged in set to sage-5.13.rc0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:50 Changed 4 years ago by leif

  • Commit 7547fceeb85afc0b6b932cb2487e61fb94fef405 deleted
  • Status changed from closed to needs_work

For the record, according to http://trac.macports.org/ticket/41033#comment:24, the cdefs.h bug has been fixed in Xcode 5.1.

Furthermore, there's no check whether the file exists at all, and set -e is used...

comment:51 follow-up: Changed 4 years ago by vbraun

cdefs.h should be part of the commandline tools. The only bug is that we don't diagnose the absence of the command line tools in a better way. Also, this ticket is closed. Open a new one.

comment:52 in reply to: ↑ 51 Changed 4 years ago by leif

Replying to vbraun:

cdefs.h should be part of the commandline tools. The only bug is that we don't diagnose the absence of the command line tools in a better way.

Yeah, and it probably should be checked whether the fix is still necessary.

Also, this ticket is closed.

Really?

Open a new one.

Selber.

comment:53 Changed 4 years ago by leif

P.S.: The top-level configure should check for the presence of Xcode's command line tools on Darwin, shouldn't it?

Note: See TracTickets for help on using tickets.