Upgrade cvxopt in sage from 0.9 to 1.1.2

[was]

We are shipping an *ancient* version of cvxopt in sage. It's worse than Debian shipping sage-3.0.5!

Note that upgrading cvxopt should be fairly easy, since basically nothing in Sage depends on cvxopt.

[wdj]

Fly in the ointment: The recent version has this statement on its license page  http://abel.ee.ucla.edu/cvxopt/copyright.html:

CVXOPT is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

Older versions apparently did not have that.

[schilly]

I've recently upgraded it for me to 1.1.2. The problem is, that the sources from them did not compile on my ubuntu 9.10 machine. So I went to the  ubuntu packges db for lucid and grabbed their version. I don't know what they did in their patch, but I guess it's non trivial...

My spkg is  here.

The only remaining modification I had to made to run the examples from the cvxopt website in %python mode was to replace

from random import ...

to

from sage.misc.prandom import ...

in ./src/src/python/__init__.py at several places.

Q:

1. what's the usual/best mechanism to avoid using Sage's random and switch back to python's random?!
2. i have no idea what the solaris patches did in the older version, neither do i know how to get it building on another system :(

[schilly]

I've created an updated p1 spkg.

Using the 1.1.2 sources directly, I get this error site-packages/cvxopt/base.so: undefined symbol: _g95_stop_blank ... I also fiddled around with the setup.py file.

 cvxopt 1.1.2 p1 spkg is here

[mvngu]

Ticket #1620 is a duplicate of this one.

[dimpase]

Replying to schilly:

I've created an updated p1 spkg. Using the 1.1.2 sources directly, I get this error site-packages/cvxopt/base.so: undefined symbol: _g95_stop_blank ... I also fiddled around with the setup.py file.  cvxopt 1.1.2 p1 spkg is here

sage -t "devel/sage-work/sage/numerical/optimize.py"

bumps out essentially due to:

sage: from cvxopt import cholmod
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/home/dima/sage/devel/sage-work/sage/<ipython console> in <module>()

ImportError: /home/dima/sage/local/lib/python2.6/site-packages/cvxopt/cholmod.so: undefined symbol: _g95_filename
sage: 

Can you reproduce this on a stand-alone build of cvxopt?

[dimpase]

patch for cvxopt-1.1.2.p1/src/src/setup.py

[dimpase]

Replying to schilly:

I've created an updated p1 spkg. Using the 1.1.2 sources directly, I get this error site-packages/cvxopt/base.so: undefined symbol: _g95_stop_blank ... I also fiddled around with the setup.py file.  cvxopt 1.1.2 p1 spkg is here

please check the patch I just uploaded. It fixes this problem; you just had to link against more dynamic libs...

[schilly]

Replying to dimpase:

please check the patch I just uploaded. It fixes this problem; you just had to link against more dynamic libs...

Thanks, I knew that it is something with that! Works now!!! ;)

I've uploaded  1.1.1.p2 here

Next I'll try if using the debian/ubuntu version of it was really necessary.

For everyone who wants to try this, don't forget that you have to disable the preparser in Sage via preparser(False) ... otherwise there are unknown types when you try to create a matrix with cvxopt's matrix command.

[dimpase]

Replying to schilly:

I've uploaded  1.1.1.p2 here Next I'll try if using the debian/ubuntu version of it was really necessary.

please take out f77blas all over in setup.py, for this is apparently obsolete and not needed - and also nukes the installation on Mac OS X (ppc) -- (otherwise it works on the latter platform)

[dimpase]

Replying to schilly:

I think that really what remains to be done is to remove dependencies on obsolete fortrans (f77), see my other comment on this. Let's get it done!

Replying to dimpase:

please check the patch I just uploaded. It fixes this problem; you just had to link against more dynamic libs...

Thanks, I knew that it is something with that! Works now!!! ;) I've uploaded  1.1.1.p2 here Next I'll try if using the debian/ubuntu version of it was really necessary.

I don't see a necessity to try non-debian/ubunty version. If you look at the debian patches, you see that all they changed is the source were slight tweaks in setup.py

For everyone who wants to try this, don't forget that you have to disable the preparser in Sage via preparser(False) ... otherwise there are unknown types when you try to create a matrix with cvxopt's matrix command.

Does this mean that we should think of importing cvxopt's matrix into Sage under some other name? I don't know what the usual Sage's way to deal with such things, i.e. name clashes between packages, is.

[drkirkby]

• Have the license issue been resolved?
• Has this been tested on Solaris?

[dimpase]

Replying to drkirkby:

* Have the license issue been resolved?

it is GPL v3 or later. Does it matter? I suppose I can ask the authors to tweak it, if it is really necessary.

* Has this been tested on Solaris?

no, but I can try on one of Skynet's machines (perhaps you can tell me which one is most likely to work :)), or you can try it yourself. (I don't have a ready Solaris install anywhere).

[drkirkby]

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

sage: license()

says all code except jsmath is GPL2, and apparently jsmath is ok, as Sage does not link to it.

The code will not build on skynet, as there are no SPARC machines there. It should build it on 't2' easily though.

I suggest you download sage 4.3.0.1 from one of the mirrors

http://www.sagemath.org/download-solaris.html

Use the following settings.

kirkby@t2:[~] $ echo $PATH
/usr/local/gcc-4.4.1-sun-linker/bin:/usr/local/bin2:/usr/bin:/usr/ccs/bin:/usr/local/bin:/usr/sfw/bin:/bin:/usr/sbin
kirkby@t2:[~] $ echo $LD_LIBRARY_PATH
/usr/local/gcc-4.4.1-sun-linker/lib:=/usr/local/gcc-4.4.1-sun-linker/lib/sparcv9:/usr/local/lib

type make, and build Sage, then try your package.

There is also a binary of Sage on the mirrors. You could download that. I'm not precisely sure what you then need to do to build just your package using the binary as a starting point.

The latest Sage source will not build on Solaris, but 4.3.0.1 will.

Dave

[dimpase]

Replying to drkirkby:

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

Well, cvxopt is an optional package, so it must be in the same boat as jmath, or some gap packages, that are also not GPL 2.

{{{ sage: license() }}} says all code except jsmath is GPL2, and apparently jsmath is ok, as Sage does not link to it. The code will not build on skynet, as there are no SPARC machines there. It should build it on 't2' easily though.

hmm, isn't this a sparc/solaris?

SunOS mark2 5.10 Generic_127111-01 sun4u sparc SUNW,Sun-Blade-2500

Dima

[drkirkby]

Replying to dimpase:

Replying to drkirkby:

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

Well, cvxopt is an optional package, so it must be in the same boat as jmath, or some gap packages, that are also not GPL 2.

Fair enough.

{{{ sage: license() }}} says all code except jsmath is GPL2, and apparently jsmath is ok, as Sage does not link to it. The code will not build on skynet, as there are no SPARC machines there. It should build it on 't2' easily though.

hmm, isn't this a sparc/solaris? SunOS mark2 5.10 Generic_127111-01 sun4u sparc SUNW,Sun-Blade-2500 Dima

Yes, it is. Sorry, I was not aware of the existance of that machine.

However, I do not know how the compilers and paths are configured on that machine. You need to have GNU make & GNU tar in your path before the Sun ones, and you need to have the Sun linker (/usr/ccs/bin/ld) in your path before any GNU ones. There are some general instructions on building Sage on Solaris at  http://wiki.sagemath.org/solaris which you would need to follow.

I've written some somewhat simpler instructions at  http://wiki.sagemath.org/devel/Building-Sage-on-the-T5240-t2 on how to build Sage on 't2'. They are simpler, as I have already put the right tools in the right locations.

The Sun Blade 2500 (mark2) should be quicker than the T5240 (t2) at building Sage. However it would require some setting up of the build environment to build Sage. If you just want an easy solution, 't2' will just work, albeit not as quickly as the Sun Blade 2500.

Dave

[dimpase]

Replying to drkirkby:

hmm, isn't this a sparc/solaris? SunOS mark2 5.10 Generic_127111-01 sun4u sparc SUNW,Sun-Blade-2500 Dima

Yes, it is. Sorry, I was not aware of the existance of that machine. However, I do not know how the compilers and paths are configured on that machine. You need to have GNU make & GNU tar in your path before the Sun ones, and you need to have the Sun linker (/usr/ccs/bin/ld) in your path before any GNU ones. There are some general instructions on building Sage on Solaris at  http://wiki.sagemath.org/solaris which you would need to follow.

well, on Skynet there is /usr/local/skynet_bash_profile that you can source upon login (from .bashrc, or just manually), and this gives you the ready setup to build Sage.

I don't have an account on t2, it seems to me. By the way, absent-mindedly I started building 4.3.3.alpha on there, and it went till gnutls, where it stopped... I noticed that gnutls is over 2 years old, version 2.2.2, whereas the current one is 2.8.5. Shouldn't one upgrade to this one, before even trying to fix this?

Dima

[dimpase]

Replying to drkirkby:

You better check the license issue with William. Code should be GPL 2 or GPL2+, but there are exceptions if a package is optional and some other conditions - I've never fully understood under what conditions code can be GLP 3. But you might find you can only use the latest version which is GPL 2, and not a GPL 3 version.

I emailed William about this, and he said it's OK in this case, it can be v3.

[drkirkby]

Replying to dimpase:

Replying to drkirkby:

However, I do not know how the compilers and paths are configured on that machine. You need to have GNU make & GNU tar in your path before the Sun ones, and you need to have the Sun linker (/usr/ccs/bin/ld) in your path before any GNU ones. There are some general instructions on building Sage on Solaris at  http://wiki.sagemath.org/solaris which you would need to follow.

well, on Skynet there is /usr/local/skynet_bash_profile that you can source upon login (from .bashrc, or just manually), and this gives you the ready setup to build Sage.

I'm not however aware of anyone building Sage on there recently, so I don't know if the environment is set up suitably. Quite a few Solaris-specific changes have been made in the last year, and some of them might not be compatible with the build system on there. I don't know. Specifically, if

gcc -v 

shows gcc was configured with the GNU linker, then the GNU linker must be in your path before the Sun linker. (Basically, whatever linker gcc uses, must be in your path first, as some code makes the assumption the first linker in your path is the one gcc uses, which might not be true.) I'm not aware of a foolproof test of this.

There should be something like

--with-ld=/usr/ccs/bin/ld

if the Sun linker was used, or

--with-ld=/path/to/gnu/ld

if the GNU linker was used.

Certainly, I've never had a problem with gnutls failing on 't2'.

I don't have an account on t2, it seems to me.

I've emailed William to ask if you can have an account on t2, as that might be the simplest solution, though that Blade 2500 would be significantly faster than 't2'.

By the way, absent-mindedly I started building 4.3.3.alpha on there, and it went till gnutls, where it stopped... I noticed that gnutls is over 2 years old, version 2.2.2, whereas the current one is 2.8.5. Shouldn't one upgrade to this one, before even trying to fix this? Dima

I can see your point about upgrading, though I'm not aware of any particular issues with the version in Sage. That version will build on Solaris. You could try appending /usr/sfw/lib to your LD_LIBRARY_PATH. I have known of issues with gnutls on OpenSolaris?, but not on Solaris 10.

I see your post about the GPL 3. That bit is sorted out then.

Dave