Opened 8 years ago

Closed 8 years ago

Last modified 6 years ago

#14505 closed enhancement (fixed)

add experimental package csdp

Reported by: dimpase Owned by: tbd
Priority: major Milestone: sage-5.11
Component: packages: experimental Keywords: semidefinite programming
Cc: ppurka, vbraun, mehdi.ghasemi Merged in: sage-5.11.rc0
Authors: Dmitrii Pasechnik Reviewers: Punarbasu Purkayastha, Nathann Cohen
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by leif)

CSDP is a fast semidefinite programming solver, one of the very best around, written in C, with a callable shared library https://projects.coin-or.org/Csdp/ (unfortunately is has a GPL-incompatible licence, which means it cannot be made standard package)

This ticket adds CSDP: a stand-alone executable, a libtool-made shared library, and headers, to Sage. One interesting software component, using Sage and CSDP, is Flagmatic : http://flagmatic.org/

The modifications to CSDP needed for Sage support, basically, libtoolization and the spkg infrastructure, is hosted here: https://github.com/dimpase/csdp

A Cython interface will be added on a separate ticket.

The spkg is here: http://boxen.math.washington.edu/home/dima/packages/csdp-6.2.spkg

Apply to $SAGE_LOCAL/bin/

Attachments (1)

trac_14505.patch (384 bytes) - added by dimpase 8 years ago.
sage-scripts patch

Download all attachments as: .zip

Change History (61)

comment:1 Changed 8 years ago by dimpase

  • Description modified (diff)
  • Status changed from new to needs_review

comment:2 follow-up: Changed 8 years ago by ncohen

HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much :-/

Nathann

comment:3 in reply to: ↑ 2 ; follow-up: Changed 8 years ago by dimpase

Replying to ncohen:

HMmmm... So you did modify the sources to get this to work ? Last time I checked the Sage guys did not like that very much :-/

Nathann

How the hell one is supposed to libtoolize the thing otherwise? That's the only modification (and some header layout was changed to conform to autotools standards). I also wanted to stay with git rather than using svn, so I imported the original svn repo. (and the latter was not changed for, like, 6 years, you know...)

I hope that my changes will be accepted by upstream, eventually. It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Last edited 8 years ago by dimpase (previous) (diff)

comment:4 Changed 8 years ago by dimpase

I added a bit more to SPKG.txt and updated the spkg accordingly (i.e. only SPKG.txt was changed).

comment:5 in reply to: ↑ 3 ; follow-up: Changed 8 years ago by leif

Replying to dimpase:

It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)

comment:6 follow-up: Changed 8 years ago by leif

So here's a pull request:

  • spkg-install

    diff --git a/spkg-install b/spkg-install
    a b  
    1717echo "Configuring csdp..."
    1818./configure --prefix="$SAGE_LOCAL"
    1919if [ $? -ne 0 ]; then
    20    echo "Error configuring csdp."
     20   echo >&2 "Error configuring csdp."
    2121   exit 1
    2222fi
    2323

B)

Last edited 8 years ago by leif (previous) (diff)

comment:7 in reply to: ↑ 5 Changed 8 years ago by dimpase

Replying to leif:

Replying to dimpase:

It is certainly possible to create the cumulative diff of my changes against the original source, I just don't see the point of this, given that all this can be extracted from the public git repo the source lives in.

Because this is the way how Sage development is supposed to happen (especially the review process)... ;-)

But this is kind of stupid... What actually makes sense to review, my autotools config files, are going to be buried under lots of automatically generated files. And, strictly speaking, I might even choose not to include them into the spkg :–)

comment:8 in reply to: ↑ 6 Changed 8 years ago by dimpase

Replying to leif:

So here's a pull request:

Thanks! Fixed and updated the spkg.

comment:9 follow-up: Changed 8 years ago by leif

Well, probably consider everything in src/ your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.

comment:10 in reply to: ↑ 9 Changed 8 years ago by dimpase

Replying to leif:

Well, probably consider everything in src/ your upstream; everything else (i.e., the "Sage part") should still be tracked separately, within the spkg, subject to get reviewed on Trac.

well, I just followed the example of libGAP spkg.

comment:11 follow-up: Changed 8 years ago by ppurka

@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.

comment:12 in reply to: ↑ 11 ; follow-ups: Changed 8 years ago by dimpase

Replying to ppurka:

@dimpase You can add the configure.ac and other stuff as a patch file, right? Then the src/ folder will be exactly as upstream.

in this case you need autotools, and Sage doesn't have them all.

comment:13 in reply to: ↑ 12 ; follow-up: Changed 8 years ago by ppurka

Replying to dimpase:

in this case you need autotools, and Sage doesn't have them all.

I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:

$ tar xf csdp-source.tbz2
$ cp -a csdp-source{,.new}
$ cd csdp-source.new && # create your configure.ac
$ ./autoreconf etc etc
$ cd ..
$ diff -Naur csdp-source csdp-source.new > csdp.patch

And then use the patch file to patch the src/ directory.

comment:14 in reply to: ↑ 13 ; follow-up: Changed 8 years ago by leif

Replying to ppurka:

Replying to dimpase:

in this case you need autotools, and Sage doesn't have them all.

I mean. Generate all the extra files you want to generate and create a single diff file which you can attach to the spkg. Something like this:

$ tar xf csdp-source.tbz2
$ cp -a csdp-source{,.new}
$ cd csdp-source.new && # create your configure.ac
$ ./autoreconf etc etc
$ cd ..
$ diff -Naur csdp-source csdp-source.new > csdp.patch

And then use the patch file to patch the src/ directory.

... making sure that configure etc. are newer than the files they're generated from.

However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/ isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.

comment:15 in reply to: ↑ 14 Changed 8 years ago by dimpase

  • Cc vbraun added

Replying to leif:

However, if upstream's (almost) dead, simply treating Dima's fork as upstream is probably better. But even in that case, everything outside src/ isn't upstream, no matter whether upstream happens to be a Sage developer, so should properly get tracked by Sage, in a separate, independent repo.

As I already said, libGAP spkg does not follow the latter "requirement":

$ pwd
/usr/local/src/sage/sage-5.9.rc0/spkg/standard/libgap-4.5.7.p2
$ hg log
changeset:   1:c5d34c333de4
tag:         tip
user:        Jeroen Demeyer <jdemeyer@cage.ugent.be>
date:        Thu Mar 28 19:21:53 2013 +0000
summary:     Added tag libgap-4.5.7.p2 for changeset c034d0063012

changeset:   0:c034d0063012
tag:         libgap-4.5.7.p2
user:        Volker Braun <vbraun@stp.dias.ie>
date:        Thu Feb 28 13:10:53 2013 -1000
summary:     This is not the true repository, read SPKG.txt

$ 

I don't see why you'd like to disallow me doing the same here.

Last edited 8 years ago by dimpase (previous) (diff)

comment:16 Changed 8 years ago by vbraun

LibGAP will eventually also become an independent "upstream" and release tarballs, with a more conventional spkg. The reason for the current state is just that it took years of fiddling and trying different approaches to make the thing work. It was definitely not clear from the outset how it can be made to work, or whether it is appropriate to shoehorn it into a spkg at all. But over time I realize that tracking libgap as a patch on a subset of the GAP distribution with some scripts that munge source code is the right way.

comment:17 follow-up: Changed 8 years ago by leif

$SAGE_LOCAL/bin/csdp needs to get added to .hgignore (in the same folder).

Changed 8 years ago by dimpase

sage-scripts patch

comment:18 Changed 8 years ago by dimpase

  • Description modified (diff)

comment:19 in reply to: ↑ 17 Changed 8 years ago by dimpase

Replying to leif:

$SAGE_LOCAL/bin/csdp needs to get added to .hgignore (in the same folder).

done.

comment:20 Changed 8 years ago by dimpase

Last edited 8 years ago by dimpase (previous) (diff)

comment:21 Changed 8 years ago by leif

  • Work issues set to ignore

comment:22 Changed 8 years ago by leif

  • Work issues ignore deleted

comment:23 Changed 8 years ago by leif

  • Description modified (diff)

comment:24 Changed 8 years ago by leif

  • Description modified (diff)

comment:25 follow-up: Changed 8 years ago by dimpase

updated spkg with proper hg history http://sage.math.washington.edu/home/dima/packages/csdp-6.2.spkg

All the changes to SPKG.txt, spkg-install, etc (but not in src/) are properly tracked - well there is one more extra commit on top, moving files to the proper place, but I hope this is OK.

The history extraction is automatic, done by hg convert extension, see https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Last edited 8 years ago by dimpase (previous) (diff)

comment:26 Changed 8 years ago by leif

  • Description modified (diff)

Updated the spkg URL in the description.

comment:27 in reply to: ↑ 25 Changed 8 years ago by dimpase

  • Cc mehdi.ghasemi added

Replying to dimpase:

I believe I have addressed all the criticism, please have a look.

comment:28 in reply to: ↑ 12 ; follow-up: Changed 8 years ago by jdemeyer

As for the question of upstream: I have created a new precedent with lrcalc. I am shipping changes to autotools source files (e.g. configure.ac) as a patch, but I apply the patch and run autotools at packaging time with a spkg-src script. Even though src/ is not shipped as upstream, the changes are clearly documented (by the spkg-src script and a patch to be applied at packaging time).

A slightly weaker form of this is used for the PARI spkg: there I am adding some auto-generated files to the src/ directory (but no patch before packaging).

comment:29 in reply to: ↑ 28 ; follow-up: Changed 8 years ago by dimpase

Replying to jdemeyer:

As for the question of upstream: I have created a new precedent with lrcalc. I am shipping changes to autotools source files (e.g. configure.ac) as a patch...

Aren't we moving away from plain patches towards more sanity?

I hope do not need to resort to do more shell script programming in order to patchify the setup, for this means doing poor man revision *control* instead of the proper (and present) one. Yes, I can create a git mega-patch and apply it to the old revision at the spkg buildtime, but I don't see the benefit of this - to the contrary, it's much less clear this way.

I am already doing more than is done in libGAP: I am shipping the full history of the spkg infrastructure, i.e. of SPKG.txt and spkg-install. spkg is created fully automatically, from a github repo, with all the changes preserved. So one can see my changes to the original upstream (changed the layout of headers and added autotools). And, as I mentioned, the latter hasn't been changed in 6 years...

comment:30 in reply to: ↑ 29 Changed 8 years ago by jdemeyer

Replying to dimpase:

Replying to jdemeyer:

As for the question of upstream: I have created a new precedent with lrcalc. I am shipping changes to autotools source files (e.g. configure.ac) as a patch...

Aren't we moving away from plain patches towards more sanity?

It seems that what you are doing with csdp isn't much different from what I am doing with lrcalc, see comment 28. The only real difference is that I don't host anything on github but all code is inside the spkg itself.

comment:31 Changed 8 years ago by ncohen

Sooooooooooooooooooo what is missing in this patch before csdp can be included ? It is still in needs_review, is there any reason to set it back to needs_work ? O_o

Nathann

comment:32 Changed 8 years ago by ppurka

What is needs work here? It is only needs review. I am fine with the changes, but haven't been able to look at it in detail.

comment:33 Changed 8 years ago by vbraun

  • Reviewers set to Punarbasu Purkayastha

Excellent, at least we have a volunteer ;-)

comment:34 follow-up: Changed 8 years ago by ncohen

Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.

Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.

There is no problem with using commands like git or anything else at this level, since it is only run by the guy who builds the SPKG, not by everybody who installs Sage.

Nathann

comment:35 Changed 8 years ago by ncohen

(by the way, for some reason there is no problem at all with tar on my laptop, but my office's computer refuses to extract it. Weird O_o)

Nathann

comment:36 follow-up: Changed 8 years ago by ppurka

I will be trying out some LP bounds soon, so I will get to look into this package. :-)

@ncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?

comment:37 in reply to: ↑ 36 Changed 8 years ago by ncohen

@ncohen - that's very little information. What OS and what version of the OS are you running? What options are you giving to tar? Is your tar symlinked to busybox (very unlikely), for instance?

Oh sorry. I had sent a couple of emails about that to Dima, that's why I was so vague.

I just don't get what is happening. We checked that my version of the spkg had the right md5sum (*cfcba3f), but when I run

tmp$ tar xjvf csdp-6.2.spkg 

nothing at all happens. It just waits forever. And here's the tar I use

~$ tar --version
tar (GNU tar) 1.26

But well.. O_o

Nathann

comment:38 Changed 8 years ago by ncohen

Okayyyyyyyyyyyyyy... Turns out there was something wrong with some automounting stuff I failed to properly install a week ago. Now it doesn't hang anymore. Mystery cleared : everything that is automatic and/or is remotely related to windows fs should never be installed >_<

Nathann

comment:39 Changed 8 years ago by ppurka

Is your tar aliased to something? Can you run

\tar xjvf csdp-6.2.spkg

yes, with the \.

comment:40 Changed 8 years ago by ppurka

Oh ok. It is all fixed for you, then it's good. Yeah, sometimes it is better to avoid mixing windows stuff with linux stuff. Things break down in mysterious ways.

comment:41 in reply to: ↑ 34 ; follow-up: Changed 8 years ago by dimpase

Replying to ncohen:

Well. If you say that this thing has not been updated for years, I guess that modifying its sources to make it run is not that bad ... This being said I haven't been able to find a mention of CSDP 6.2 on internet, only 6.1.1 seems to be available on the original website.

svn repo has 6.2 (beta); here is this commit imported: https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a

Could you also explain somewhere how this spkg can be built from scratch ? I remember having seen somewhere in a spkg a "create spkg" file with an internal variable equal to the url of the .tar.gz file, and running that ended up in creating the SPKG itself.

this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

comment:42 in reply to: ↑ 41 ; follow-up: Changed 8 years ago by ncohen

Yo !

svn repo has 6.2 (beta); here is this commit imported: https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a

Ahahah. Cool :-D

this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh script but it ends up like that :

/tmp/csdp/spkg$ ./make-spkg.sh 
make: *** No rule to make target `dist'.  Stop.

Annnnd I have no idea if what I do is what I should to build the spkg :-)

Nathann

comment:43 in reply to: ↑ 42 Changed 8 years ago by dimpase

Replying to ncohen:

Yo !

svn repo has 6.2 (beta); here is this commit imported: https://github.com/dimpase/csdp/commit/77081795bfe14c1440581dfa839fd9bb6026736a

Ahahah. Cool :-D

this is the script that does the job (on the full git repo): https://github.com/dimpase/csdp/blob/master/spkg/make-spkg.sh

Should I mention this in SPKG.txt ?

Yep, it would be nice ! I just downloaded your git repository and ran this make-spkg.sh script but it ends up like that :

/tmp/csdp/spkg$ ./make-spkg.sh 
make: *** No rule to make target `dist'.  Stop.

Annnnd I have no idea if what I do is what I should to build the spkg :-)

right, you need to populate the tree with makefiles first, by ./configure at the top level.

Nathann

comment:44 follow-up: Changed 8 years ago by ncohen

O_o

There is no configure file in at the top level O_o

/tmp/csdp/spkg$ ls
convert_mapfile  install.sh  LICENSE  make-spkg.sh  README.md  spkg-install  SPKG.txt
/tmp/csdp/spkg$ cd ..
/tmp/csdp$ ./configure
bash: ./configure: No such file or directory

Nathann

comment:45 in reply to: ↑ 44 Changed 8 years ago by dimpase

Replying to ncohen:

O_o

There is no configure file in at the top level O_o

Oops, sorry.

Welcome to the autoHell :-)

I've added the detailed instructions on bilding the spkg to the SPKG.txt and updated the spkg file. Hopefully it's clear now.

comment:46 follow-up: Changed 8 years ago by ppurka

There is still something missing. I had to add the line

convert =

in my [extensions] section of ~/.hgrc. Even so, it fails with the following error on running make-spkg.

8 added support to make Sage spkg, and the corresponding files
7 explained NOSHORTS thing in SPKG.txt
6 Update README.md
5 added missing >&2
4 further rectifications of NOSHORTS and SPKG.txt
3 an attempt to use hg convert to fake the history in spkg/
2 tweaking "hg convert" thing
1 yet more tweaks to make_spkg
0 added instructions to build spkg
updating bookmarks
mv: cannot stat ‘csdp-hg/.hg’: No such file or directory

comment:47 in reply to: ↑ 46 Changed 8 years ago by dimpase

Replying to ppurka:

There is still something missing. I had to add the line

convert =

in my [extensions] section of ~/.hgrc.

OK, I've updated SPKG.txt mentioning this, as well as git and autotools/libtool requirements for spkg creating.

Even so, it fails with the following error on running make-spkg.

weird. Do have have an old/super-new hg?

Does it create csdp-hg/ in some other place for you?

(I presume you do ./make-spkg.sh)

comment:48 follow-up: Changed 8 years ago by ppurka

There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh

/tmp/csdpnew» sage -sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/mnt/usb/Installations/sage-5.10.rc0
/tmp/csdpnew» hg --version
Mercurial Distributed SCM (version 2.2.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/tmp/csdpnew» which hg
/mnt/usb/Installations/sage-5.10.rc0/local/bin/hg
/tmp/csdpnew»
Exited Sage subshell.
/tmp/csdpnew» hg --version
Mercurial Distributed SCM (version 2.4.2)
(see http://mercurial.selenic.com for more information)

Copyright (C) 2005-2012 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/tmp/csdpnew» which hg
/usr/bin/hg
/tmp/csdpnew» ls
aclocal.m4	config.guess   config.sub	depcomp  install-sh  m4		    matlab     sage	 theta
AUTHORS		config.h       configure	doc	 lib	     Makefile	    missing    solver
autom4te.cache	config.h.in    configure.ac	example  libtool     Makefile.am    NEWS       spkg
ChangeLog	config.log     COPYING		include  LICENSE     Makefile.in    README     stamp-h1
compile		config.status  csdp-6.2.tar.gz	INSTALL  ltmain.sh   Makefile.orig  README.md  test
/tmp/csdpnew» cd spkg
/tmp/csdpnew/spkg» ls
convert_mapfile  csdp-6.2  install.sh  LICENSE	make-spkg.sh  README.md  spkg-install  SPKG.txt
/tmp/csdpnew/spkg» cd -
/tmp/csdpnew
/tmp/csdpnew» find . -name 'csdp-hg'
/tmp/csdpnew»

comment:49 in reply to: ↑ 48 Changed 8 years ago by dimpase

Replying to ppurka:

There is no csdp-hg anywhere and I suppose it is using sage's hg when I am running ./make-spkg.sh

/tmp/csdpnew» sage -sh

things in /tmp might be a bit funny...

Also, do you have anything unusual in ~/.hgrc ? I imagine hg-git might interfere...

comment:50 follow-up: Changed 8 years ago by ppurka

No. Nothing special in that file:

[ui]
username = Punarbasu Purkayastha <ppurka snip...>

[extensions]
rebase =
mq =
fetch =
pager =
graphlog =
color =
record =
hgext.extdiff =
convert =

[diff]
git = true

[extdiff]
cmd.merge_hg =

[merge-tools]
merge_hg.args = "$local" "$other" "$base"

[pager]
pager = less
attend = annotate, cat, diff, log, glog, qdiff, export

[alias]
mylog = glog -l15 --template '\033[34;40m{rev} \033[00;40m- \033[01;33;40m{tags} \033[0;37;40m{desc|firstline|fill68} \033[0;32;40m{date|age}\033[0;37;40m <{author|person}> \033[37;40m \033[35;40m{branches}\033[37;40m\n'
l = mylog

Last edited 8 years ago by ppurka (previous) (diff)

comment:51 in reply to: ↑ 50 ; follow-up: Changed 8 years ago by dimpase

Replying to ppurka:

did you try building outside /tmp ? I am unable to reproduce your issue.

comment:52 in reply to: ↑ 51 Changed 8 years ago by ppurka

Replying to dimpase:

did you try building outside /tmp ? I am unable to reproduce your issue.

That is indeed very weird. It works outside /tmp. Must be some quirk of hg.

comment:53 Changed 8 years ago by ppurka

How do I test the installation? I think there should be something to test the installation of this package, without having to go through pycsdp or flagmatic.

I tried the example.c file in example directory, but the compilation seems to work only if I actually compiled csdp and compiled example.c from within the sources directory.

comment:54 Changed 8 years ago by ncohen

Well, if the binary is correctly build and runs, I would say that the spkg does its job :-)

You can set this ticket to positive review from my behalf too once that you will be satisfied with what it does :-)

Nathann

comment:55 Changed 8 years ago by ppurka

  • Reviewers changed from Punarbasu Purkayastha to Punarbasu Purkayastha, Nathann Cohen
  • Status changed from needs_review to positive_review

The binary does build and run. And I have tested the compiled binay with flagmatic earlier. Since you are ok with it, let's give it a +1. ;-)

comment:56 Changed 8 years ago by schilly

spkg is on the server.

comment:57 Changed 8 years ago by jdemeyer

  • Merged in set to sage-5.11.rc0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:58 Changed 7 years ago by kcrisman

Possible followup: #17391.

comment:59 Changed 6 years ago by was

Any interest in moving this to optional? I have user requests for this on SageMathCloud?. E.g.,

Dear SageMath,

I encountered a problem at 

The package csdp (used by lovasz_theta() function) is not installed. The 
current error message is somewhat misleading, as it tells me to run 'sage 
-i csdp'. Doing so fails due to insufficient permissions.

NOTE: The latter thing about "insufficient permissions" is yet another way that we need to distinguish between system-wide and private sage installs.

comment:60 Changed 6 years ago by dimpase

I am all for it.

Note: See TracTickets for help on using tickets.