add experimental package csdp

[dimpase]

CSDP is a fast semidefinite programming solver, one of the very best around, written in C, with a callable shared library ​https://projects.coin-or.org/Csdp/ (unfortunately is has a GPL-incompatible licence, which means it cannot be made standard package)

This ticket adds CSDP: a stand-alone executable, a libtool-made shared library, and headers, to Sage. One interesting software component, using Sage and CSDP, is Flagmatic : ​http://flagmatic.org/

The modifications to CSDP needed for Sage support, basically, libtoolization and the spkg infrastructure, is hosted here: ​https://github.com/dimpase/csdp

A Cython interface will be added on a separate ticket.

The spkg is here: ​http://boxen.math.washington.edu/home/dima/packages/csdp-6.2.spkg

Apply to $SAGE_LOCAL/bin/

• trac_14505.patch​

[ncohen]

HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much :-/

Nathann

[dimpase]

Replying to ncohen:

HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much :-/

Nathann

How the hell one is supposed to libtoolize the thing otherwise? That's the only modification (and some header layout was changed to conform to autotools standards). I also wanted to stay with git rather than using svn, so I imported the original svn repo. (and the latter was not changed for, like, 6 years, you know...)

I hope that my changes will be accepted by upstream, eventually. It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

[dimpase]

I added a bit more to SPKG.txt and updated the spkg accordingly (i.e. only SPKG.txt was changed).

[leif]

Replying to dimpase:

It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)

[leif]

So here's a pull request:

spkg-install

@@ -17 +17 @@
 echo "Configuring csdp..."
 ./configure --prefix="$SAGE_LOCAL"
 if [ $? -ne 0 ]; then
-   echo "Error configuring csdp."
+   echo >&2 "Error configuring csdp."
    exit 1
 fi
 

B)

[dimpase]

Replying to leif:

Replying to dimpase:

It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)

But this is kind of stupid... What actually makes sense to review, my autotools config files, are going to be buried under lots of automatically generated files. And, strictly speaking, I might even choose not to include them into the spkg :–)

[dimpase]

Replying to leif:

So here's a pull request:

Thanks! Fixed and updated the spkg.

[leif]

Well, probably consider everything in src/ your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.

[dimpase]

Replying to leif:

Well, probably consider everything in src/ your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.

well, I just followed the example of libGAP spkg.

[ppurka]

@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.

[dimpase]

Replying to ppurka:

@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.

in this case you need autotools, and Sage doesn't have them all.

[ppurka]

Replying to dimpase:

in this case you need autotools, and Sage doesn't have them all.

I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:

$ tar xf csdp-source.tbz2
$ cp -a csdp-source{,.new}
$ cd csdp-source.new && # create your configure.ac
$ ./autoreconf etc etc
$ cd ..
$ diff -Naur csdp-source csdp-source.new > csdp.patch

And then use the patch file to patch the src/ directory.

[leif]

Replying to ppurka:

Replying to dimpase:

in this case you need autotools, and Sage doesn't have them all.

I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:

$ tar xf csdp-source.tbz2
$ cp -a csdp-source{,.new}
$ cd csdp-source.new && # create your configure.ac
$ ./autoreconf etc etc
$ cd ..
$ diff -Naur csdp-source csdp-source.new > csdp.patch

And then use the patch file to patch the src/ directory.

... making sure that configure etc. are newer than the files they're generated from.

However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/ isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.

[dimpase]

Replying to leif:

However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/ isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.

As I already said, libGAP spkg does not follow the latter "requirement":

$ pwd
/usr/local/src/sage/sage-5.9.rc0/spkg/standard/libgap-4.5.7.p2
$ hg log
changeset:   1:c5d34c333de4
tag:         tip
user:        Jeroen Demeyer <jdemeyer@cage.ugent.be>
date:        Thu Mar 28 19:21:53 2013 +0000
summary:     Added tag libgap-4.5.7.p2 for changeset c034d0063012

changeset:   0:c034d0063012
tag:         libgap-4.5.7.p2
user:        Volker Braun <vbraun@stp.dias.ie>
date:        Thu Feb 28 13:10:53 2013 -1000
summary:     This is not the true repository, read SPKG.txt

$ 

I don't see why you'd like to disallow me doing the same here.

[vbraun]

LibGAP will eventually also become an independent "upstream" and release tarballs, with a more conventional spkg. The reason for the current state is just that it took years of fiddling and trying different approaches to make the thing work. It was definitely not clear from the outset how it can be made to work, or whether it is appropriate to shoehorn it into a spkg at all. But over time I realize that tracking libgap as a patch on a subset of the GAP distribution with some scripts that munge source code is the right way.

[leif]

$SAGE_LOCAL/bin/csdp needs to get added to .hgignore (in the same folder).

[dimpase]

sage-scripts patch

[dimpase]

Replying to leif:

$SAGE_LOCAL/bin/csdp needs to get added to .hgignore (in the same folder).

done.

[dimpase]

updated spkg with proper hg history ​http://sage.math.washington.edu/home/dima/packages/csdp-6.2.spkg

All the changes to SPKG.txt, spkg-install, etc (but not in src/) are properly tracked - well there is one more extra commit on top, moving files to the proper place, but I hope this is OK.

The history extraction is automatic, done by hg convert extension, see ​https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

[leif]

Updated the spkg URL in the description.

[dimpase]

Replying to dimpase:

I believe I have addressed all the criticism, please have a look.

[jdemeyer]

As for the question of upstream: I have created a new precedent with lrcalc. I am shipping changes to autotools source files (e.g. configure.ac) as a patch, but I apply the patch and run autotools at packaging time with a spkg-src script. Even though src/ is not shipped as upstream, the changes are clearly documented (by the spkg-src script and a patch to be applied at packaging time).

A slightly weaker form of this is used for the PARI spkg: there I am adding some auto-generated files to the src/ directory (but no patch before packaging).

[dimpase]

Replying to jdemeyer:

As for the question of upstream: I have created a new precedent with lrcalc. I am shipping changes to autotools source files (e.g. configure.ac) as a patch...

Aren't we moving away from plain patches towards more sanity?

I hope do not need to resort to do more shell script programming in order to patchify the setup, for this means doing poor man revision *control* instead of the proper (and present) one. Yes, I can create a git mega-patch and apply it to the old revision at the spkg buildtime, but I don't see the benefit of this - to the contrary, it's much less clear this way.

I am already doing more than is done in libGAP: I am shipping the full history of the spkg infrastructure, i.e. of SPKG.txt and spkg-install. spkg is created ​fully automatically, from a github repo, with all the changes preserved. So one can see my changes to the original upstream (changed the layout of headers and added autotools). And, as I mentioned, the latter hasn't been changed in 6 years...

[jdemeyer]

Replying to dimpase:

Replying to jdemeyer:

As for the question of upstream: I have created a new precedent with lrcalc. I am shipping changes to autotools source files (e.g. configure.ac) as a patch...

Aren't we moving away from plain patches towards more sanity?

It seems that what you are doing with csdp isn't much different from what I am doing with lrcalc, see comment 28. The only real difference is that I don't host anything on github but all code is inside the spkg itself.

[ncohen]

Sooooooooooooooooooo what is missing in this patch before csdp can be included ? It is still in needs_review, is there any reason to set it back to needs_work ? O_o

Nathann

[ppurka]

What is needs work here? It is only needs review. I am fine with the changes, but haven't been able to look at it in detail.

[vbraun]

Excellent, at least we have a volunteer ;-)

[ncohen]

Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.

Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.

There is no problem with using commands like git or anything else at this level, since it is only run by the guy who builds the SPKG, not by everybody who installs Sage.

Nathann

[ncohen]

(by the way, for some reason there is no problem at all with tar on my laptop, but my office's computer refuses to extract it. Weird O_o)

Nathann

[ppurka]

I will be trying out some LP bounds soon, so I will get to look into this package. :-)

@ncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?

[ncohen]

@ncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?

Oh sorry. I had sent a couple of emails about that to Dima, that's why I was so vague.

I just don't get what is happening. We checked that my version of the spkg had the right md5sum (*cfcba3f), but when I run

tmp$ tar xjvf csdp-6.2.spkg 

nothing at all happens. It just waits forever. And here's the tar I use

~$ tar --version
tar (GNU tar) 1.26

But well.. O_o

Nathann

[ncohen]

Okayyyyyyyyyyyyyy... Turns out there was something wrong with some automounting stuff I failed to properly install a week ago. Now it doesn't hang anymore. Mystery cleared : everything that is automatic and/or is remotely related to windows fs should never be installed >_<

Nathann

[ppurka]

Is your tar aliased to something? Can you run

\tar xjvf csdp-6.2.spkg

yes, with the \.

[ppurka]

Oh ok. It is all fixed for you, then it's good. Yeah, sometimes it is better to avoid mixing windows stuff with linux stuff. Things break down in mysterious ways.

[dimpase]

Replying to ncohen:

Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.

svn repo has 6.2 (beta); here is this commit imported: ​https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a

Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.

this is the script that does the job (on the full git repo): ​https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

[ncohen]

Yo !

svn repo has 6.2 (beta); here is this commit imported: ​https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a

Ahahah. Cool :-D

this is the script that does the job (on the full git repo): ​https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh script but it ends up like that :

/tmp/csdp/spkg$ ./make-spkg.sh 
make: *** No rule to make target `dist'.  Stop.

Annnnd I have no idea if what I do is what I should to build the spkg :-)

Nathann

[dimpase]

Replying to ncohen:

Yo !

svn repo has 6.2 (beta); here is this commit imported: ​https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a

Ahahah. Cool :-D

this is the script that does the job (on the full git repo): ​https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh script but it ends up like that :

/tmp/csdp/spkg$ ./make-spkg.sh 
make: *** No rule to make target `dist'.  Stop.

Annnnd I have no idea if what I do is what I should to build the spkg :-)

right, you need to populate the tree with makefiles first, by ./configure at the top level.

Nathann

[ncohen]

O_o

There is no configure file in at the top level O_o

/tmp/csdp/spkg$ ls
convert_mapfile  install.sh  LICENSE  make-spkg.sh  README.md  spkg-install  SPKG.txt
/tmp/csdp/spkg$ cd ..
/tmp/csdp$ ./configure
bash: ./configure: No such file or directory

Nathann

[dimpase]

Replying to ncohen:

O_o

There is no configure file in at the top level O_o

Oops, sorry.

Welcome to the autoHell :-)

I've added the detailed instructions on bilding the spkg to the ​SPKG.txt and updated the spkg file. Hopefully it's clear now.

[ppurka]

There is still something missing. I had to add the line

convert =

in my [extensions] section of ~/.hgrc. Even so, it fails with the following error on running make-spkg.

8 added support to make Sage spkg, and the corresponding files
7 explained NOSHORTS thing in SPKG.txt
6 Update README.md
5 added missing >&2
4 further rectifications of NOSHORTS and SPKG.txt
3 an attempt to use hg convert to fake the history in spkg/
2 tweaking "hg convert" thing
1 yet more tweaks to make_spkg
0 added instructions to build spkg
updating bookmarks
mv: cannot stat ‘csdp-hg/.hg’: No such file or directory

[dimpase]

Replying to ppurka:

There is still something missing. I had to add the line

convert =

in my [extensions] section of ~/.hgrc.

OK, I've updated SPKG.txt mentioning this, as well as git and autotools/libtool requirements for spkg creating.

Even so, it fails with the following error on running make-spkg.

weird. Do have have an old/super-new hg?

Does it create csdp-hg/ in some other place for you?

(I presume you do ./make-spkg.sh)

[ppurka]

There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh

/tmp/csdpnew» sage -sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/mnt/usb/Installations/sage-5.10.rc0
/tmp/csdpnew» hg --version
Mercurial Distributed SCM (version 2.2.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/tmp/csdpnew» which hg
/mnt/usb/Installations/sage-5.10.rc0/local/bin/hg
/tmp/csdpnew»
Exited Sage subshell.
/tmp/csdpnew» hg --version
Mercurial Distributed SCM (version 2.4.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/tmp/csdpnew» which hg
/usr/bin/hg
/tmp/csdpnew» ls
aclocal.m4	config.guess   config.sub	depcomp  install-sh  m4		    matlab     sage	 theta
AUTHORS		config.h       configure	doc	 lib	     Makefile	    missing    solver
autom4te.cache	config.h.in    configure.ac	example  libtool     Makefile.am    NEWS       spkg
ChangeLog	config.log     COPYING		include  LICENSE     Makefile.in    README     stamp-h1
compile		config.status  csdp-6.2.tar.gz	INSTALL  ltmain.sh   Makefile.orig  README.md  test
/tmp/csdpnew» cd spkg
/tmp/csdpnew/spkg» ls
convert_mapfile  csdp-6.2  install.sh  LICENSE	make-spkg.sh  README.md  spkg-install  SPKG.txt
/tmp/csdpnew/spkg» cd -
/tmp/csdpnew
/tmp/csdpnew» find . -name 'csdp-hg'
/tmp/csdpnew»

[dimpase]

Replying to ppurka:

There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh

/tmp/csdpnew» sage -sh

things in /tmp might be a bit funny...

Also, do you have anything unusual in ~/.hgrc ? I imagine hg-git might interfere...

[ppurka]

No. Nothing special in that file:

[ui]
username = Punarbasu Purkayastha <ppurka snip...>

[extensions]
rebase =
mq =
fetch =
pager =
graphlog =
color =
record =
hgext.extdiff =
convert =

[diff]
git = true

[extdiff]
cmd.merge_hg =

[merge-tools]
merge_hg.args = "$local" "$other" "$base"

[pager]
pager = less
attend = annotate, cat, diff, log, glog, qdiff, export

[alias]
mylog = glog -l15 --template '\033[34;40m{rev} \033[00;40m- \033[01;33;40m{tags} \033[0;37;40m{desc|firstline|fill68} \033[0;32;40m{date|age}\033[0;37;40m <{author|person}> \033[37;40m \033[35;40m{branches}\033[37;40m\n'
l = mylog

[dimpase]

Replying to ppurka:

did you try building outside /tmp ? I am unable to reproduce your issue.

[ppurka]

Replying to dimpase:

did you try building outside /tmp ? I am unable to reproduce your issue.

That is indeed very weird. It works outside /tmp. Must be some quirk of hg.

[ppurka]

How do I test the installation? I think there should be something to test the installation of this package, without having to go through pycsdp or flagmatic.

I tried the example.c file in example directory, but the compilation seems to work only if I actually compiled csdp and compiled example.c from within the sources directory.

[ncohen]

Well, if the binary is correctly build and runs, I would say that the spkg does its job :-)

You can set this ticket to positive review from my behalf too once that you will be satisfied with what it does :-)

Nathann

[ppurka]

The binary does build and run. And I have tested the compiled binay with flagmatic earlier. Since you are ok with it, let's give it a +1. ;-)

[schilly]

spkg is on the server.

[kcrisman]

Possible followup: #17391.

[was]

Any interest in moving this to optional? I have user requests for this on SageMathCloud?. E.g.,

Dear SageMath,

I encountered a problem at 

The package csdp (used by lovasz_theta() function) is not installed. The 
current error message is somewhat misleading, as it tells me to run 'sage 
-i csdp'. Doing so fails due to insufficient permissions.

NOTE: The latter thing about "insufficient permissions" is yet another way that we need to distinguish between system-wide and private sage installs.

[dimpase]

I am all for it.