#14505 closed enhancement (fixed)
add experimental package csdp
Reported by: | dimpase | Owned by: | tbd |
---|---|---|---|
Priority: | major | Milestone: | sage-5.11 |
Component: | packages: experimental | Keywords: | semidefinite programming |
Cc: | ppurka, vbraun, mehdi.ghasemi | Merged in: | sage-5.11.rc0 |
Authors: | Dmitrii Pasechnik | Reviewers: | Punarbasu Purkayastha, Nathann Cohen |
Report Upstream: | N/A | Work issues: | |
Branch: | Commit: | ||
Dependencies: | Stopgaps: |
Description (last modified by )
CSDP is a fast semidefinite programming solver, one of the very best around, written in C, with a callable shared library https://projects.coin-or.org/Csdp/ (unfortunately is has a GPL-incompatible licence, which means it cannot be made standard package)
This ticket adds CSDP: a stand-alone executable, a libtool-made shared library, and headers, to Sage. One interesting software component, using Sage and CSDP, is Flagmatic : http://flagmatic.org/
The modifications to CSDP needed for Sage support, basically, libtoolization and the spkg infrastructure, is hosted here: https://github.com/dimpase/csdp
A Cython interface will be added on a separate ticket.
The spkg is here: http://boxen.math.washington.edu/home/dima/packages/csdp-6.2.spkg
Apply to $SAGE_LOCAL/bin/
Attachments (1)
Change History (61)
comment:1 Changed 8 years ago by
- Description modified (diff)
- Status changed from new to needs_review
comment:2 follow-up: ↓ 3 Changed 8 years ago by
comment:3 in reply to: ↑ 2 ; follow-up: ↓ 5 Changed 8 years ago by
Replying to ncohen:
HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much
:-/
Nathann
How the hell one is supposed to libtoolize the thing otherwise? That's the only modification (and some header layout was changed to conform to autotools standards). I also wanted to stay with git rather than using svn, so I imported the original svn repo. (and the latter was not changed for, like, 6 years, you know...)
I hope that my changes will be accepted by upstream, eventually. It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.
comment:4 Changed 8 years ago by
I added a bit more to SPKG.txt and updated the spkg accordingly (i.e. only SPKG.txt was changed).
comment:5 in reply to: ↑ 3 ; follow-up: ↓ 7 Changed 8 years ago by
Replying to dimpase:
It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.
Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)
comment:6 follow-up: ↓ 8 Changed 8 years ago by
So here's a pull request:
-
spkg-install
diff --git a/spkg-install b/spkg-install
a b 17 17 echo "Configuring csdp..." 18 18 ./configure --prefix="$SAGE_LOCAL" 19 19 if [ $? -ne 0 ]; then 20 echo "Error configuring csdp."20 echo >&2 "Error configuring csdp." 21 21 exit 1 22 22 fi 23 23
B)
comment:7 in reply to: ↑ 5 Changed 8 years ago by
Replying to leif:
Replying to dimpase:
It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.
Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)
But this is kind of stupid... What actually makes sense to review, my autotools config files, are going to be buried under lots of automatically generated files. And, strictly speaking, I might even choose not to include them into the spkg :–)
comment:8 in reply to: ↑ 6 Changed 8 years ago by
comment:9 follow-up: ↓ 10 Changed 8 years ago by
Well, probably consider everything in src/
your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.
comment:10 in reply to: ↑ 9 Changed 8 years ago by
Replying to leif:
Well, probably consider everything in
src/
your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.
well, I just followed the example of libGAP spkg.
comment:11 follow-up: ↓ 12 Changed 8 years ago by
@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.
comment:12 in reply to: ↑ 11 ; follow-ups: ↓ 13 ↓ 28 Changed 8 years ago by
Replying to ppurka:
@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.
in this case you need autotools, and Sage doesn't have them all.
comment:13 in reply to: ↑ 12 ; follow-up: ↓ 14 Changed 8 years ago by
Replying to dimpase:
in this case you need autotools, and Sage doesn't have them all.
I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:
$ tar xf csdp-source.tbz2 $ cp -a csdp-source{,.new} $ cd csdp-source.new && # create your configure.ac $ ./autoreconf etc etc $ cd .. $ diff -Naur csdp-source csdp-source.new > csdp.patch
And then use the patch file to patch the src/ directory.
comment:14 in reply to: ↑ 13 ; follow-up: ↓ 15 Changed 8 years ago by
Replying to ppurka:
Replying to dimpase:
in this case you need autotools, and Sage doesn't have them all.
I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:
$ tar xf csdp-source.tbz2 $ cp -a csdp-source{,.new} $ cd csdp-source.new && # create your configure.ac $ ./autoreconf etc etc $ cd .. $ diff -Naur csdp-source csdp-source.new > csdp.patchAnd then use the patch file to patch the src/ directory.
... making sure that configure
etc. are newer than the files they're generated from.
However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/
isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.
comment:15 in reply to: ↑ 14 Changed 8 years ago by
- Cc vbraun added
Replying to leif:
However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside
src/
isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.
As I already said, libGAP spkg does not follow the latter "requirement":
$ pwd /usr/local/src/sage/sage-5.9.rc0/spkg/standard/libgap-4.5.7.p2 $ hg log changeset: 1:c5d34c333de4 tag: tip user: Jeroen Demeyer <jdemeyer@cage.ugent.be> date: Thu Mar 28 19:21:53 2013 +0000 summary: Added tag libgap-4.5.7.p2 for changeset c034d0063012 changeset: 0:c034d0063012 tag: libgap-4.5.7.p2 user: Volker Braun <vbraun@stp.dias.ie> date: Thu Feb 28 13:10:53 2013 -1000 summary: This is not the true repository, read SPKG.txt $
I don't see why you'd like to disallow me doing the same here.
comment:16 Changed 8 years ago by
LibGAP will eventually also become an independent "upstream" and release tarballs, with a more conventional spkg. The reason for the current state is just that it took years of fiddling and trying different approaches to make the thing work. It was definitely not clear from the outset how it can be made to work, or whether it is appropriate to shoehorn it into a spkg at all. But over time I realize that tracking libgap as a patch on a subset of the GAP distribution with some scripts that munge source code is the right way.
comment:17 follow-up: ↓ 19 Changed 8 years ago by
$SAGE_LOCAL/bin/csdp
needs to get added to .hgignore
(in the same folder).
comment:18 Changed 8 years ago by
- Description modified (diff)
comment:19 in reply to: ↑ 17 Changed 8 years ago by
comment:20 Changed 8 years ago by
comment:21 Changed 8 years ago by
- Work issues set to ignore
comment:22 Changed 8 years ago by
- Work issues ignore deleted
comment:23 Changed 8 years ago by
- Description modified (diff)
comment:24 Changed 8 years ago by
- Description modified (diff)
comment:25 follow-up: ↓ 27 Changed 8 years ago by
updated spkg with proper hg history http://sage.math.washington.edu/home/dima/packages/csdp-6.2.spkg
All the changes to SPKG.txt, spkg-install, etc (but not in src/) are properly tracked - well there is one more extra commit on top, moving files to the proper place, but I hope this is OK.
The history extraction is automatic, done by hg convert
extension, see
https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh
comment:26 Changed 8 years ago by
- Description modified (diff)
Updated the spkg URL in the description.
comment:27 in reply to: ↑ 25 Changed 8 years ago by
- Cc mehdi.ghasemi added
Replying to dimpase:
I believe I have addressed all the criticism, please have a look.
comment:28 in reply to: ↑ 12 ; follow-up: ↓ 29 Changed 8 years ago by
As for the question of upstream: I have created a new precedent with lrcalc
. I am shipping changes to autotools source files (e.g. configure.ac
) as a patch, but I apply the patch and run autotools at packaging time with a spkg-src
script. Even though src/
is not shipped as upstream, the changes are clearly documented (by the spkg-src
script and a patch to be applied at packaging time).
A slightly weaker form of this is used for the PARI spkg: there I am adding some auto-generated files to the src/
directory (but no patch before packaging).
comment:29 in reply to: ↑ 28 ; follow-up: ↓ 30 Changed 8 years ago by
Replying to jdemeyer:
As for the question of upstream: I have created a new precedent with
lrcalc
. I am shipping changes to autotools source files (e.g. configure.ac) as a patch...
Aren't we moving away from plain patches towards more sanity?
I hope do not need to resort to do more shell script programming in order to patchify the setup, for this means doing poor man revision *control* instead of the proper (and present) one. Yes, I can create a git mega-patch and apply it to the old revision at the spkg buildtime, but I don't see the benefit of this - to the contrary, it's much less clear this way.
I am already doing more than is done in libGAP: I am shipping the full history of the spkg infrastructure, i.e. of SPKG.txt and spkg-install. spkg is created fully automatically, from a github repo, with all the changes preserved. So one can see my changes to the original upstream (changed the layout of headers and added autotools). And, as I mentioned, the latter hasn't been changed in 6 years...
comment:30 in reply to: ↑ 29 Changed 8 years ago by
Replying to dimpase:
Replying to jdemeyer:
As for the question of upstream: I have created a new precedent with
lrcalc
. I am shipping changes to autotools source files (e.g. configure.ac) as a patch...Aren't we moving away from plain patches towards more sanity?
It seems that what you are doing with csdp isn't much different from what I am doing with lrcalc
, see comment 28. The only real difference is that I don't host anything on github but all code is inside the spkg itself.
comment:31 Changed 8 years ago by
Sooooooooooooooooooo what is missing in this patch before csdp can be included ? It is still in needs_review, is there any reason to set it back to needs_work ? O_o
Nathann
comment:32 Changed 8 years ago by
What is needs work here? It is only needs review. I am fine with the changes, but haven't been able to look at it in detail.
comment:33 Changed 8 years ago by
- Reviewers set to Punarbasu Purkayastha
Excellent, at least we have a volunteer ;-)
comment:34 follow-up: ↓ 41 Changed 8 years ago by
Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.
Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.
There is no problem with using commands like git or anything else at this level, since it is only run by the guy who builds the SPKG, not by everybody who installs Sage.
Nathann
comment:35 Changed 8 years ago by
(by the way, for some reason there is no problem at all with tar on my laptop, but my office's computer refuses to extract it. Weird O_o
)
Nathann
comment:36 follow-up: ↓ 37 Changed 8 years ago by
I will be trying out some LP bounds soon, so I will get to look into this package. :-)
@ncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?
comment:37 in reply to: ↑ 36 Changed 8 years ago by
@ncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?
Oh sorry. I had sent a couple of emails about that to Dima, that's why I was so vague.
I just don't get what is happening. We checked that my version of the spkg had the right md5sum (*cfcba3f), but when I run
tmp$ tar xjvf csdp-6.2.spkg
nothing at all happens. It just waits forever. And here's the tar I use
~$ tar --version tar (GNU tar) 1.26
But well.. O_o
Nathann
comment:38 Changed 8 years ago by
Okayyyyyyyyyyyyyy... Turns out there was something wrong with some automounting stuff I failed to properly install a week ago. Now it doesn't hang anymore. Mystery cleared : everything that is automatic and/or is remotely related to windows fs should never be installed >_<
Nathann
comment:39 Changed 8 years ago by
Is your tar aliased to something? Can you run
\tar xjvf csdp-6.2.spkg
yes, with the \
.
comment:40 Changed 8 years ago by
Oh ok. It is all fixed for you, then it's good. Yeah, sometimes it is better to avoid mixing windows stuff with linux stuff. Things break down in mysterious ways.
comment:41 in reply to: ↑ 34 ; follow-up: ↓ 42 Changed 8 years ago by
Replying to ncohen:
Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.
svn repo has 6.2 (beta); here is this commit imported: https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a
Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.
this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh
Should I mention this in SPKG.txt
?
comment:42 in reply to: ↑ 41 ; follow-up: ↓ 43 Changed 8 years ago by
Yo !
svn repo has 6.2 (beta); here is this commit imported: https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a
Ahahah. Cool :-D
this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh
Should I mention this in
SPKG.txt
?
Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh
script but it ends up like that :
/tmp/csdp/spkg$ ./make-spkg.sh make: *** No rule to make target `dist'. Stop.
Annnnd I have no idea if what I do is what I should to build the spkg :-)
Nathann
comment:43 in reply to: ↑ 42 Changed 8 years ago by
Replying to ncohen:
Yo !
svn repo has 6.2 (beta); here is this commit imported: https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a
Ahahah. Cool
:-D
this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh
Should I mention this in
SPKG.txt
?Yep, it would be nice ! I just downloaded your git repository and ran this
make-spkg.sh
script but it ends up like that :/tmp/csdp/spkg$ ./make-spkg.sh make: *** No rule to make target `dist'. Stop.Annnnd I have no idea if what I do is what I should to build the spkg
:-)
right, you need to populate the tree with makefiles first, by ./configure at the top level.
Nathann
comment:44 follow-up: ↓ 45 Changed 8 years ago by
O_o
There is no configure file in at the top level O_o
/tmp/csdp/spkg$ ls convert_mapfile install.sh LICENSE make-spkg.sh README.md spkg-install SPKG.txt /tmp/csdp/spkg$ cd .. /tmp/csdp$ ./configure bash: ./configure: No such file or directory
Nathann
comment:45 in reply to: ↑ 44 Changed 8 years ago by
comment:46 follow-up: ↓ 47 Changed 8 years ago by
There is still something missing. I had to add the line
convert =
in my [extensions]
section of ~/.hgrc
. Even so, it fails with the following error on running make-spkg
.
8 added support to make Sage spkg, and the corresponding files 7 explained NOSHORTS thing in SPKG.txt 6 Update README.md 5 added missing >&2 4 further rectifications of NOSHORTS and SPKG.txt 3 an attempt to use hg convert to fake the history in spkg/ 2 tweaking "hg convert" thing 1 yet more tweaks to make_spkg 0 added instructions to build spkg updating bookmarks mv: cannot stat ‘csdp-hg/.hg’: No such file or directory
comment:47 in reply to: ↑ 46 Changed 8 years ago by
Replying to ppurka:
There is still something missing. I had to add the line
convert =in my
[extensions]
section of~/.hgrc
.
OK, I've updated SPKG.txt mentioning this, as well as git and autotools/libtool requirements for spkg creating.
Even so, it fails with the following error on running
make-spkg
.
weird. Do have have an old/super-new hg?
Does it create csdp-hg/ in some other place for you?
(I presume you do ./make-spkg.sh)
comment:48 follow-up: ↓ 49 Changed 8 years ago by
There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh
/tmp/csdpnew» sage -sh Starting subshell with Sage environment variables set. Don't forget to exit when you are done. Beware: * Do not do anything with other copies of Sage on your system. * Do not use this for installing Sage packages using "sage -i" or for running "make" at Sage's root directory. These should be done outside the Sage shell. Bypassing shell configuration files... Note: SAGE_ROOT=/mnt/usb/Installations/sage-5.10.rc0 /tmp/csdpnew» hg --version Mercurial Distributed SCM (version 2.2.2) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. /tmp/csdpnew» which hg /mnt/usb/Installations/sage-5.10.rc0/local/bin/hg /tmp/csdpnew» Exited Sage subshell. /tmp/csdpnew» hg --version Mercurial Distributed SCM (version 2.4.2) (see http://mercurial.selenic.com for more information) Copyright (C) 2005-2012 Matt Mackall and others This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. /tmp/csdpnew» which hg /usr/bin/hg /tmp/csdpnew» ls aclocal.m4 config.guess config.sub depcomp install-sh m4 matlab sage theta AUTHORS config.h configure doc lib Makefile missing solver autom4te.cache config.h.in configure.ac example libtool Makefile.am NEWS spkg ChangeLog config.log COPYING include LICENSE Makefile.in README stamp-h1 compile config.status csdp-6.2.tar.gz INSTALL ltmain.sh Makefile.orig README.md test /tmp/csdpnew» cd spkg /tmp/csdpnew/spkg» ls convert_mapfile csdp-6.2 install.sh LICENSE make-spkg.sh README.md spkg-install SPKG.txt /tmp/csdpnew/spkg» cd - /tmp/csdpnew /tmp/csdpnew» find . -name 'csdp-hg' /tmp/csdpnew»
comment:49 in reply to: ↑ 48 Changed 8 years ago by
Replying to ppurka:
There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running
./make-spkg.sh
/tmp/csdpnew» sage -sh
things in /tmp
might be a bit funny...
Also, do you have anything unusual in ~/.hgrc
?
I imagine hg-git might interfere...
comment:50 follow-up: ↓ 51 Changed 8 years ago by
No. Nothing special in that file:
[ui] username = Punarbasu Purkayastha <ppurka snip...> [extensions] rebase = mq = fetch = pager = graphlog = color = record = hgext.extdiff = convert = [diff] git = true [extdiff] cmd.merge_hg = [merge-tools] merge_hg.args = "$local" "$other" "$base" [pager] pager = less attend = annotate, cat, diff, log, glog, qdiff, export [alias] mylog = glog -l15 --template '\033[34;40m{rev} \033[00;40m- \033[01;33;40m{tags} \033[0;37;40m{desc|firstline|fill68} \033[0;32;40m{date|age}\033[0;37;40m <{author|person}> \033[37;40m \033[35;40m{branches}\033[37;40m\n' l = mylog
comment:51 in reply to: ↑ 50 ; follow-up: ↓ 52 Changed 8 years ago by
Replying to ppurka:
did you try building outside /tmp
?
I am unable to reproduce your issue.
comment:52 in reply to: ↑ 51 Changed 8 years ago by
Replying to dimpase:
did you try building outside
/tmp
? I am unable to reproduce your issue.
That is indeed very weird. It works outside /tmp
. Must be some quirk of hg.
comment:53 Changed 8 years ago by
How do I test the installation? I think there should be something to test the installation of this package, without having to go through pycsdp or flagmatic.
I tried the example.c file in example directory, but the compilation seems to work only if I actually compiled csdp and compiled example.c from within the sources directory.
comment:54 Changed 8 years ago by
Well, if the binary is correctly build and runs, I would say that the spkg does its job :-)
You can set this ticket to positive review from my behalf too once that you will be satisfied with what it does :-)
Nathann
comment:55 Changed 8 years ago by
- Reviewers changed from Punarbasu Purkayastha to Punarbasu Purkayastha, Nathann Cohen
- Status changed from needs_review to positive_review
The binary does build and run. And I have tested the compiled binay with flagmatic earlier. Since you are ok with it, let's give it a +1. ;-)
comment:56 Changed 7 years ago by
spkg is on the server.
comment:57 Changed 7 years ago by
- Merged in set to sage-5.11.rc0
- Resolution set to fixed
- Status changed from positive_review to closed
comment:58 Changed 6 years ago by
Possible followup: #17391.
comment:59 Changed 5 years ago by
Any interest in moving this to optional? I have user requests for this on SageMathCloud?. E.g.,
Dear SageMath, I encountered a problem at The package csdp (used by lovasz_theta() function) is not installed. The current error message is somewhat misleading, as it tells me to run 'sage -i csdp'. Doing so fails due to insufficient permissions.
NOTE: The latter thing about "insufficient permissions" is yet another way that we need to distinguish between system-wide and private sage installs.
comment:60 Changed 5 years ago by
I am all for it.
HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much
:-/
Nathann