Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#8172 closed enhancement (fixed)

Support for CPLEX

Reported by: ncohen Owned by: jkantor
Priority: major Milestone: sage-4.4.1
Component: numerical Keywords:
Cc: malb, schilly Merged in: sage-4.4.1.alpha2
Authors: Nathann Cohen Reviewers: Dmitrii Pasechnik, Martin Albrecht
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Description (last modified by drkirkby)

This patch enables support for Cplex, when the new package #8171 for CBC is installed. It also includes several lines of documentation to explain how CPLEX can be used, and fixes a bad docstring which I noticed while testing this package :-)

Nathann does have access to 't2'. Minh has made it clear the last alpha (4.3.4.alpha1) builds ok on 't2', so testing this should be possible for Nathann.

Attachments (2)

trac_8172-updated.patch (34.3 KB) - added by dimpase 9 years ago.
fixed few places where "Mixed Integer" was omitted
trac_8172.patch (34.3 KB) - added by ncohen 9 years ago.

Download all attachments as: .zip

Change History (41)

comment:1 Changed 9 years ago by ncohen

  • Status changed from new to needs_review

comment:2 Changed 9 years ago by malb

You are doctesting solve_coin in solve_cplex

comment:3 Changed 9 years ago by ncohen

OOps :-)

The code from mip_cplex is a pure copy of mip_coin ( I only needed to change the types from OsiCbcSolverInterface? to OsiCplexSolverInterface?, etc .... )

This code has already been reviewed, by the way.....

Fixed ! Thank you :-)

Nathann

comment:4 Changed 9 years ago by malb

I wonder if we can factor out the differences and this way would make it very very easy to hook up new solvers (with say, some function pointers and such) The redundancy of copying a function does not seem very elegant to me.

comment:5 follow-up: Changed 9 years ago by ncohen

I do not like it either, but this way it answers two other difficulties :

  • Coin me be compiled independently from the presence of CPLEX. So I can not be assume in mip_coin that CPLEX libraries are available and that this file *can* actually be compiled.
  • Testing for the presence of a CPLEX solver amounts to try to import solve_cplex, which is nice and can only be done if the file has been compiled successfully

Nathann

comment:6 in reply to: ↑ 5 Changed 9 years ago by malb

We could still reduce the amount of code duplication. Say we replace the functions solve_* by instances of a class OSISolver which would get function pointers pointing to the correct function to call for each solver and have a call method which implements the current solve_* function. So for each new solver we'd only need to add a new instance. All the points you raise above would be addressed.

comment:7 Changed 9 years ago by ncohen

Hmm.. It is almost done, but I have a few problems with cimporting in mip_cplex the c_solve function defined in mip_coin.... :-)

comment:8 Changed 9 years ago by ncohen

Ok, I have spent 3 or 4 hours dealing with this bug, if you want to give it a try it is almost finished : we but have to find a way to import this * function...

I'm sorry for my rudeness, but I can not stand these stupid error messages anymore !

Nathann

comment:9 Changed 9 years ago by ncohen

If you can fix it, I will obviously finish to clean the patch....

Nathann

comment:10 Changed 9 years ago by drkirkby

  • Status changed from needs_review to needs_info

What platforms has this been tested on? It needs to work on OS X, Linux and Solaris. It's not clear from reading this that anyone has actually checked it works on Solaris.

There's general information about building on Solaris at

http://wiki.sagemath.org/solaris

Information specifically for 't2' at

http://wiki.sagemath.org/devel/Building-Sage-on-the-T5240-t2

Both the source (4.3.0.1 is the latest to build on Solaris) and a binary which will run on any SPARC can be found at http://www.sagemath.org /download-source.html

Dave

comment:11 Changed 9 years ago by ncohen

  • Status changed from needs_info to needs_review

Here is a new *WORKING* version of the patch :-)

I received plenty of help, and it can finally be used !!

I can not test this piece of code on all these platforms. Anyway, CPLEX will be used by very few users of Sage as it already requires one to have a licence, which is pretty expensive ! ;-)

Nathann

comment:12 Changed 9 years ago by ncohen

  • Status changed from needs_review to needs_work

comment:13 Changed 9 years ago by ncohen

  • Status changed from needs_work to needs_review

New version of this patch...

  • It now handles exceptions
  • I added options to have Cbc work multithread when the user wants it
  • I refactored some code which is now much more readable
  • I rounded some values in the docstrings, as Cplex behaves a bit differently from it friends concerning numerical noise

This patch can now be reviewed, and will be a nice addition whether one uses Cplex or not because of the multithread ! :-)

Nathann

comment:14 Changed 9 years ago by ncohen

Cplex seems to be freely available for academics and students :

http://www.ibm.com/developerworks/forums/thread.jspa?threadID=319342&tstart=0

Nathann

comment:15 Changed 9 years ago by drkirkby

  • Status changed from needs_review to needs_info

I understand you feel this would be useful, especially given CPLEX is available free for academics and students. (I'm neither a student or academic, so can't get a copy free myself). Given CPLEX is available for OSX, Solaris on both SPARC and x86, I believe it would be wise that you test it on those platforms.

You do have access to 't2' (Solaris) and 'bsd' (OS X) I assume.

Testing on Solaris or OS X is not necessary for an experimental package (I'm not sure about optional), but certainly it would never be accepted as standard if it did not build on Solaris or OS X. I'm not quite sure where this package aims to fit (standard/experiemental/optional). That would dictate whether it needs to build on Solaris and OS X.

I've put to 'need info' so you can clarify this for us.

Dave

comment:16 Changed 9 years ago by malb

  • it seems you define ctypedef struct c_OsiSolverInterface "OsiSolverInterface" a few times in a few different files. Why?
  • CPLEX has a few # optional - requires Cbc
  • Why did you change log=False to log=0? Also the docstring does not match anymore.
  • # a hand-made memset why don't you just call memset?

comment:17 Changed 9 years ago by malb

Btw. applies cleanly and builds fine on sage.math

comment:18 Changed 9 years ago by ncohen

Hello !!!

  • ctypedef repeated : fixed
  • optional cbc : fixed
  • because Cbc has several levels of output. Cplex too, perhaps, nce I will know how to tune it :-)
  • because I did not know where to find the memset function : fixed

Thank you very much for your help ! I just updated the patch :-)

Nathann

comment:19 Changed 9 years ago by ncohen

  • Status changed from needs_info to needs_review

About tests on other platforms : Cbc, GLPK and obviously CPLEX will never be made standard packages because of license incompatibilities. I am still looking for a totally free solver but haven't found one yet, nor do I hope to ever find (an efficient) one.

I am really sorry but I can not say I will find the time to test them on these platforms. I do not know them, and even if it worked I have no idea of how to make platform-dependent fixes... :-/

Nathann

comment:20 Changed 9 years ago by malb

I just ran doctests on OSX (32-bit I think, its bsd.math). I think we should only worry about the patch on all platforms. If the optional package fails on say Solaris we can deal with this later.

comment:21 Changed 9 years ago by malb

Can we allow the same number of threads parameter for CPLEX?

comment:22 Changed 9 years ago by ncohen

I have to admit I do not know... The function I use to set Coin to n threads is Coin-specific. Cplex is used through the Osi interface, which is a good way to quickly add new solvers, but also means that I can not access a solver-specific function when the interface does not let me do it.

On the other side, Cplex is not free, and the multithreaded version is even more expensive. I have no access to a multithreaded version, but I sent a request for a free licence and I am still waiting for the answer ... With a bit of luck, the licence will let me use this mode and I will be able to take a more serious look at it :-)

Nathann

comment:23 follow-up: Changed 9 years ago by malb

Dave, do you object if we give this ticket a positive review?

comment:24 in reply to: ↑ 23 Changed 9 years ago by drkirkby

Replying to malb:

Dave, do you object if we give this ticket a positive review?

Since it is optional, I think this can go without it building on Solaris. But I would draw Nathann's attention to the developers guide, in particular:

http://www.sagemath.org/doc/developer/inclusion.html?highlight=solaris

Some Sage developers are willing to help you port to OS X, Solaris and Windows. But this is no guarantee and you or your project are expected to do the heavy lifting and also support those ports upstream if there is no Sage developer who is willing to share the burden.

However, since this is not a standard package, then I think it can go.

It was not clear to me on the #8171 whether that was intended to be experimental or optional. I believe the requirements for optional are more stringent than experimental, but I don't know exactly that they are. I believe Building on Solaris and OS X might be a requirement for optional, but are certainly not for experimental. I think that point needs clarification at some point in the future.

I'm feel unable to review this, as it is outside my area, so if Micheal feels it deserves a positive review, then go ahead, but do not put my name on it.

Note also the "Author" field is not completed.

Dave

comment:25 Changed 9 years ago by drkirkby

  • Description modified (diff)

A couple of points which I believe are important before this gets a positive review.

  • The patch is based on #8171, which at the very least needs clarification as to whether it is optional or experimental. Despite being a new package, it starts with .p2. I've asked for clarification on whether #8171 is expected to be 'experimental' or 'optional'.
  • I've asked on sage-devel for clarification of whether building on Solaris is a requirement for optional packages. If so, (as I expect it is), then I believe that package (and this patch) should be checked on Solaris before getting a positive review.

comment:26 follow-up: Changed 9 years ago by ncohen

  • Authors set to ncohen

Hello !!

As Martin mentionned it on #8171, the Cbc package is to be optional, exactly as it currently is. After all, #8171 is just an update. The .p2 is present precisely because it is an update. I hesitated before writing it, though, as the version of Cbc also changed. Do we increase the p# when the version of the source is fixed, and change it otherwise ? As the source changed (I updated it), do I need to "go back" to p0 ?

To be frank, I have in the end no idea of what exactly Solaris is. I know I can check it on the internet very quickly, and this is exactly what I intend to do, but in the end this is where my knowledge of it will stop. I have absolutely no idea how to deal with it...

Nathann

comment:27 in reply to: ↑ 26 Changed 9 years ago by drkirkby

Replying to ncohen:

Hello !!

As Martin mentionned it on #8171, the Cbc package is to be optional, exactly as it currently is. After all, #8171 is just an update. The .p2 is present precisely because it is an update. I hesitated before writing it, though, as the version of Cbc also changed. Do we increase the p# when the version of the source is fixed, and change it otherwise ? As the source changed (I updated it), do I need to "go back" to p0 ?

Neither. See:

http://www.sagemath.org/doc/developer/patching_spkgs.html

If the version is changed from the upstream source code, then the new package should be called by that version number, and nothing else - not even p0. Once the first patch to that is made, it becomes p0.

To be frank, I have in the end no idea of what exactly Solaris is. I know I can check it on the internet very quickly, and this is exactly what I intend to do, but in the end this is where my knowledge of it will stop. I have absolutely no idea how to deal with it...

Nathann

Solaris is a Unix operating system. Unlike Linux, which is a Unix-like operating system, Solaris is officially classified as a POSIX compliant Unix operating system. It was created by Sun, who were recently bought by Oracle for $7 billion. Sage builds on Solaris 10 and soon will be an officially supported operating system, just as some linux distributions are.

't2', which people refer to is a Sun T5240 machine with two SPARC processors. (The SPARC processor is similar to the Intel/AMD processors, though not identical). Each processor has 64-threads (128 in total).

For any package to become a standard part of Sage, then it will need to build on Solaris 10 on SPARC. I'm unsure of the situation with optional packages. I don't mind helping if you have specific problems, but I am not going to do all the testing for you.

When I checked the IBM web site, the library was available for Solaris.

dave

comment:28 Changed 9 years ago by malb

  • Status changed from needs_review to positive_review

comment:29 Changed 9 years ago by ncohen

Hello !

I just tried to log on t2 but my password was refused. This is a bit weird as I should not even be asked for a password as I set my ssh key in authorized_keys, but it looks like this server does not know me. Is there something special to do to log on it which is different from the procedure applied for sage.math ?

Nathann

comment:30 follow-up: Changed 9 years ago by dimpase

Hi, do you understand how to request an "academic" license for CPLEX?

I went to the IBM website and registered as an academic user and downloaded the stuff, and can even compile and link CPLEX examples, but I cannot figure out where and how to get a licence (it does not come with the download...) Thanks, Dima

comment:31 in reply to: ↑ 30 Changed 9 years ago by dimpase

Replying to dimpase:

Hi, do you understand how to request an "academic" license for CPLEX?

OK, I figured it out: here are the details:

http://download.boulder.ibm.com/ibmdl/pub/software/dw/university/support/ILOGQuickStart.pdf

First, one has to register here: https://www.ibm.com/developerworks/university/software/get_software.html

and after somewhat tedious registration get to download a zip archive, that contains tarballs of several Linux, MacOS, and Solaris binary installations. Install the right tarball somewhere.

Licence can be requested here: https://www.ibm.com/developerworks/university/support/ilog.html

you will get a file named access.ilm that you e.g. can place into /usr/ilog/ilm/ (or you can go for a custom location and an environment variable to hold it (see the quickstart pdf linked above)

Dima

comment:32 follow-up: Changed 9 years ago by ncohen

From the tests done in my lab since we are using this new license ( we were using CPLEX 10 before ), this one is *much* faster, especially for integer multiflows.

Nathann

comment:33 in reply to: ↑ 32 ; follow-up: Changed 9 years ago by dimpase

Replying to ncohen:

From the tests done in my lab since we are using this new license ( we were using CPLEX 10 before ), this one is *much* faster, especially for integer multiflows.

Nathann

works for me with the free academic CPLEX license on Linux x86_64 Debian!

Changed 9 years ago by dimpase

fixed few places where "Mixed Integer" was omitted

comment:34 in reply to: ↑ 33 Changed 9 years ago by dimpase

Replying to dimpase:

Replying to ncohen:

From the tests done in my lab since we are using this new license ( we were using CPLEX 10 before ), this one is *much* faster, especially for integer multiflows.

Nathann

works for me with the free academic CPLEX license on Linux x86_64 Debian!

Good work! I am giving it a positive review, subject to the patch being updated to the one I just uploaded (that adds Mixed Integer here and there, to avoid a confusion stemming from the fact that Sage does have an LP solver, from cvxopt)

Dima

comment:35 Changed 9 years ago by ncohen

Excellent ! I looked at the result of "diff" on the two patches, and I agree with your changes ! :-)

Nathann

comment:36 Changed 9 years ago by ncohen

  • Reviewers set to dimpase

Now the two patches are the same, just to avoid confusion... ;-)

Nathann

Changed 9 years ago by ncohen

comment:37 Changed 9 years ago by was

  • Merged in set to 4.4.1.alpha2
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:38 Changed 9 years ago by mvngu

  • Authors changed from ncohen to Nathann Cohen
  • Reviewers changed from dimpase to Dmitrii Pasechnik, Martin Albrecht

comment:39 Changed 9 years ago by mvngu

  • Merged in changed from 4.4.1.alpha2 to sage-4.4.1.alpha2
Note: See TracTickets for help on using tickets.