Opened 3 years ago

Last modified 8 months ago

#22380 needs_work enhancement

Upgrade to cvxopt 1.2.x

Reported by: dimpase Owned by: embray
Priority: major Milestone: sage-8.5
Component: packages: standard Keywords: upgrade, cvxopt
Cc: arojas, embray, fbissey, gh-timokau, infinity0, pcpa, saraedum, slelievre, strogdon, thansen, tmonteil Merged in:
Authors: François Bissey Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: u/fbissey/cvxopt-1.2.2 (Commits) Commit: 87f00486d061445a8fa83ddaf9133cd7be0aaca6
Dependencies: Stopgaps:

Description (last modified by fbissey)

This ticket is to upgrade to cvxopt 1.2.2, released 2018-10-19.

Since cvxopt 1.1.9, SuiteSparse is no longer included, thus it will have to be packaged separately.

Ticket #19687 was our previous upgrade, to cvxopt 1.1.8 (released 2015-09-22).

Attachments (1)

LICENSE.txt (47.9 KB) - added by dimpase 3 years ago.
suite-sparse license

Download all attachments as: .zip

Change History (45)

Changed 3 years ago by dimpase

suite-sparse license

comment:1 Changed 3 years ago by dimpase

  • Cc fbissey added
  • Description modified (diff)

the 1st task would be to understand whether the Suite tarfile needs to be cut to exclude non-GPL compatible parts (see attachment)...

comment:2 Changed 3 years ago by fbissey

I don't know about debian and other but we split suitesparse in components in Gentoo. I would be shocked if the other didn't do the same. It looks like upstream integrated a lot of the autoconfig patches that were floating around which makes life easier. What cvxopt needs:

  • suitesparseconfig

*amd

*umfpack

*colamd

*cholmod

Indentation is for dependency. That's without optional or more complicated configuration - a minimal setup (although options using blas/lapack could be enabled without extra burden). All the above in mimimum configuration is LGPL 2.1. In non-mimimum configuration some switch to GPL-2.

Last edited 3 years ago by fbissey (previous) (diff)

comment:3 Changed 3 years ago by fbissey

From setup.py - this can all be overridden, but it horrifies me

# Guess SUITESPARSE_LIB_DIR and SUITESPARSE_INC_DIR
if platform.platform().startswith("Darwin"):
    # macOS
    SUITESPARSE_LIB_DIR = '/usr/local/lib'
    SUITESPARSE_INC_DIR = '/usr/local/include'
else: 
    # default: Ubuntu/Debian
    SUITESPARSE_LIB_DIR = "/usr/lib/x86_64-linux-gnu"
    SUITESPARSE_INC_DIR = "/usr/include/suitesparse"

The only line that is valid is the default include directory for debian. The rest is accessible by default without defining anything. Basically it destroys stuff that would work out of the box without any values. This is relevant in that it means we will have to override things manually when it could work out of the box if left alone. That's not just suitesparse. gsl, glpk... Of course there are limitations of distutils that complicate things (an empty includedir statement generate a -I in the compilation line instead of nothing at all, the compiler then gets confused. The best default is probably '.' if you have to put something).

comment:4 Changed 3 years ago by fbissey

Good news, cvxopt-1.1.9 is in gentoo now and no doctests were harmed by the upgrade in sage-on-gentoo. The bad news is that writing the spkgs for this is not high on my TODO list.

comment:5 Changed 11 months ago by slelievre

  • Cc arojas embray gh-timokau infinity0 pcpa saraedum slelievre strogdon thansen tmonteil added
  • Description modified (diff)
  • Keywords upgrade cvxopt added
  • Milestone changed from sage-7.6 to sage-wishlist
  • Summary changed from update cvxopt to 1.1.9 to Upgrade to cvxopt 1.2.1

Sage-on-Gentoo just upgraded to cvxopt 1.2.1.

See https://github.com/cschwan/sage-on-gentoo/issues/524

comment:6 follow-up: Changed 11 months ago by dimpase

It would be great if Gentoo's suite-sparse can be lifted to become a Sage package.

For starters I should get sage-on-gentoo running on my Gentoo laptop. How do I start with it?

comment:7 in reply to: ↑ 6 ; follow-up: Changed 11 months ago by fbissey

Replying to dimpase:

It would be great if Gentoo's suite-sparse can be lifted to become a Sage package.

For starters I should get sage-on-gentoo running on my Gentoo laptop. How do I start with it?

Well we are a bit old school but the instructions of the README at https://github.com/cschwan/sage-on-gentoo are still working. One day I'll update them to something a little bit more modern. Note that you also need the science overlay. I won't finish sage 8.4 tongiht I am afraid. Too much to do, it will be ready in 12 hours.

comment:8 Changed 11 months ago by Konrad127123

Note: cvxopt 1.2.2 was released a few days ago.

comment:9 Changed 11 months ago by slelievre

  • Description modified (diff)
  • Summary changed from Upgrade to cvxopt 1.2.1 to Upgrade to cvxopt 1.2.x

Updating ticket description following the release of cvxopt 1.2.2.

comment:10 in reply to: ↑ 7 Changed 11 months ago by dimpase

Replying to fbissey:

Well we are a bit old school but the instructions of the README at https://github.com/cschwan/sage-on-gentoo are still working.

A somewhat cryptic error:

!!! The ebuild selected to satisfy "sage" has unmet requirements.
- sci-mathematics/sage-8.4::sage-on-gentoo USE="X doc-html-bin -bliss -debug -doc-html -doc-pdf -doc-pdf-bin -jmol -latex -modular_decomposition -sagenb -testsuite" ABI_X86="(64)" L10N="-ca -de -en -es -fr -hu -it -ja -pt -ru -tr" PYTHON_TARGETS="python2_7 python3_6"

  The following REQUIRED_USE flag constraints are unsatisfied:
    doc-html-bin? ( l10n_en )

  The above constraints are a subset of the following complete expression:
    doc-html? ( l10n_en sagenb ) doc-pdf? ( sagenb ) testsuite? ( any-of ( doc-html doc-html-bin ) ) doc-html-bin? ( !doc-html !doc-pdf l10n_en ) doc-pdf-bin? ( !doc-html !doc-pdf l10n_en ) any-of ( python_targets_python2_7 python_targets_python3_6 )

no idea what's wrong. Changing doc-html-bin to -doc-html-bin allows the build to go forward.

comment:11 Changed 11 months ago by fbissey

That's part of the possible useflags. Basically I allow you to choose which language you want to build but because the english version is the only complete one, if you elect to have documentation, the english one is compulsory. Put

L10N="en"

in make.conf and you'll get pre-built english documentation installed.

comment:12 Changed 11 months ago by fbissey

I am going to make an attempt at a "reduced" suitesparse spkg. Reduced because I will not make default packages (which aren't used by anything else) that use cmake. Note that suitesparse will move to cmake for the whole thing in the near future.

The fun bit is that suitesparse is effectively a small mini distribution itself (but most of the upstreams are now totally absorbed by the project). It doesn't currently work very nicely with building phases unless we break down the various components. It does install directly on the file system and because it is like a mini distribution, trying to stop it will break the build system (where are my libraries and my include files).

Once it is completely cmake-ified it may actually behave better in that respect.

comment:13 Changed 11 months ago by fbissey

  • Authors set to François Bissey
  • Branch set to u/fbissey/cvxopt-1.2.2
  • Commit set to 32e028e9d3b2a048cfe8db6b56212afbfe2321c8
  • Description modified (diff)
  • Milestone changed from sage-wishlist to sage-8.5
  • Status changed from new to needs_review

SPKG.txt for suitesparse needs a bit of polish but this is a good start for anyone who wants to tests this. This is open for reviews even though I am expecting a bit more work needs to be done.


New commits:

94517d2basic suitsparse spkg
903741finitial update to cvxopt 1.2.2
093e5f3Merge branch 'develop' into cvxopt-1.22
32e028eadd suitesparse to cvxopt dependencies

comment:14 Changed 11 months ago by Konrad127123

I'm getting an Invalid checksum error for cvxopt-1.2.2.tar.gz.

cvxopt/checksums.ini says:

tarball=cvxopt-VERSION.tar.gz
sha1=725ce85fe122cd0fcda9d1b1a45ae43adf6fea2c
md5=ef9ab5f939380e9488e57b113103d96a
cksum=4099798435

but I get:

$ sha1sum ../../upstream/cvxopt-1.2.2.tar.gz 
43113a138e5e8f3ba42dc53ccb8c3c7501d5576b  ../../upstream/cvxopt-1.2.2.tar.gz
$ md5sum ../../upstream/cvxopt-1.2.2.tar.gz 
fc2546fdba2faa6c4996e9b40fce5269  ../../upstream/cvxopt-1.2.2.tar.gz
$ cksum ../../upstream/cvxopt-1.2.2.tar.gz
3288957477 1877253 ../../upstream/cvxopt-1.2.2.tar.gz

comment:15 Changed 11 months ago by fbissey

My bad I used the wrong tarball (github instead of pypi) when creating the checksum. I'll fix it shortly.

comment:16 Changed 11 months ago by git

  • Commit changed from 32e028e9d3b2a048cfe8db6b56212afbfe2321c8 to 21e7280425f0e37e07265563b97dfc527332c548

Branch pushed to git repo; I updated commit sha1. New commits:

21e7280fix cvxopt checksum

comment:17 Changed 11 months ago by git

  • Commit changed from 21e7280425f0e37e07265563b97dfc527332c548 to 87f00486d061445a8fa83ddaf9133cd7be0aaca6

Branch pushed to git repo; I updated commit sha1. New commits:

87f0048Flesh out SPKG.txt

comment:18 Changed 11 months ago by fbissey

I would be OK for things to ship as they are now but I am open to suggestions. An alternative is to split SuiteSparse into components but that's more work.

comment:19 Changed 11 months ago by Konrad127123

Building with SAGE_CHECK="yes" still fails when running headless (#24657).

comment:20 Changed 11 months ago by fbissey

I see, I was not aware of this issue. I see that matplotlib is involved, probably in the same way some doctest could be failing on debian. That will require some code inspection.

comment:21 Changed 10 months ago by dimpase

Without openmp, or with openmp installed in /usr/local/, I get I get

[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)

Is openmp a pre-req?

One way or another, the installation ought to be smarter as far as the location of openmp is concerned.

Last edited 10 months ago by dimpase (previous) (diff)

comment:22 Changed 10 months ago by fbissey

Yes it should be smarter. Now can you give me more info? Is this OS X? which sub-package of suitesparse? In short a longer log please.

comment:23 Changed 10 months ago by dimpase

This is FreeBSD 12.0. With openmp installed (in /usr/local, which is "standard" location on such a system), everything works (passes relevant Sage tests too) if I export LDFLAGS="$LDFLAGS -L/usr/local/lib" before installation. If LDFLAGS are not passed, it breaks at CHOLMOD as follows:

...
[suitesparse-5.3.0] a - cholmod_l_nesdis.o
[suitesparse-5.3.0] a - cholmod_l_camd.o
[suitesparse-5.3.0] cc -L/usr/home/dimpase/Sage/sagemath/local/lib -Wl,-rpath,/usr/home/dimpase/Sage/sagemath/local/lib  -L/usr/home/dimpase/Sage/sagemath/local/lib -L/usr/home/dimpase/Sage/sagemath/local/lib -L/usr/home/dimpase/Sage/sagemath/local/lib -shared -Wl,-soname -Wl,libcholmod.so.3 -Wl,--no-undefined -fopenmp cholmod_aat.o cholmod_add.o cholmod_band.o cholmod_change_factor.o cholmod_common.o cholmod_complex.o cholmod_copy.o cholmod_dense.o cholmod_error.o cholmod_factor.o cholmod_memory.o cholmod_sparse.o cholmod_transpose.o cholmod_triplet.o cholmod_version.o cholmod_check.o cholmod_read.o cholmod_write.o cholmod_amd.o cholmod_analyze.o cholmod_colamd.o cholmod_etree.o cholmod_factorize.o cholmod_postorder.o cholmod_rcond.o cholmod_resymbol.o cholmod_rowcolcounts.o cholmod_rowfac.o cholmod_solve.o cholmod_spsolve.o cholmod_drop.o cholmod_horzcat.o cholmod_norm.o cholmod_scale.o cholmod_sdmult.o cholmod_ssmult.o cholmod_submatrix.o cholmod_vertcat.o cholmod_symmetry.o cholmod_rowadd.o cholmod_rowdel.o cholmod_updown.o cholmod_super_numeric.o cholmod_super_solve.o cholmod_super_symbolic.o cholmod_ccolamd.o cholmod_csymamd.o cholmod_metis.o cholmod_nesdis.o cholmod_camd.o cholmod_l_aat.o cholmod_l_add.o cholmod_l_band.o cholmod_l_change_factor.o cholmod_l_common.o cholmod_l_complex.o cholmod_l_copy.o cholmod_l_dense.o cholmod_l_error.o cholmod_l_factor.o cholmod_l_memory.o cholmod_l_sparse.o cholmod_l_transpose.o cholmod_l_triplet.o cholmod_l_version.o cholmod_l_check.o cholmod_l_read.o cholmod_l_write.o cholmod_l_amd.o cholmod_l_analyze.o cholmod_l_colamd.o cholmod_l_etree.o cholmod_l_factorize.o cholmod_l_postorder.o cholmod_l_rcond.o cholmod_l_resymbol.o cholmod_l_rowcolcounts.o cholmod_l_rowfac.o cholmod_l_solve.o cholmod_l_spsolve.o cholmod_l_drop.o cholmod_l_horzcat.o cholmod_l_norm.o cholmod_l_scale.o cholmod_l_sdmult.o cholmod_l_ssmult.o cholmod_l_submatrix.o cholmod_l_vertcat.o cholmod_l_symmetry.o cholmod_l_rowadd.o cholmod_l_rowdel.o cholmod_l_updown.o cholmod_l_super_numeric.o cholmod_l_super_solve.o cholmod_l_super_symbolic.o cholmod_l_ccolamd.o cholmod_l_csymamd.o cholmod_l_metis.o cholmod_l_nesdis.o cholmod_l_camd.o -o /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12 -lm -lamd -lcolamd -lsuitesparseconfig -lccolamd -lcamd -lopenblas  -lopenblas 
[suitesparse-5.3.0] ranlib libcholmod.a
[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)
[suitesparse-5.3.0] gmake[4]: *** [Makefile:544: /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12] Error 1
[suitesparse-5.3.0] gmake[4]: Leaving directory '/usr/home/dimpase/Sage/sagemath/local/var/tmp/sage/build/suitesparse-5.3.0/src/CHOLMOD/Lib'

And with openmp not installed, it also breaks at CHOLMOD, although the output is differently ordered, somehow.

...
[suitesparse-5.3.0] a - cholmod_l_nesdis.o
[suitesparse-5.3.0] a - cholmod_l_camd.o
[suitesparse-5.3.0] ranlib libcholmod.a
[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)
[suitesparse-5.3.0] gmake[4]: *** [Makefile:544: /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12] Error 1
[suitesparse-5.3.0] gmake[4]: Leaving directory '/usr/home/dimpase/Sage/sagemath/local/var/tmp/sage/build/suitesparse-5.3.0/src/CHOLMOD/Lib'

Last edited 10 months ago by dimpase (previous) (diff)

comment:24 follow-up: Changed 10 months ago by fbissey

Yes it is unusual. And very strange. There is some control for openmp in the main makefile I think so it has to be turned off there (or autodetected properly there). But not finding stuff in /usr/local/lib is highly unusual - and it appears this is not your compiler's fault but ld's fault. Is it GNU ld, some kind of BSD ld or llvm's ld?

comment:25 Changed 10 months ago by fbissey

Is there urgency to have this in 8.5? I have been thinking that I have a longer time frame I may want to try splitting suitesparse into just the components needed for cvxopt.

comment:26 follow-up: Changed 10 months ago by dimpase

PS. I hear that openmp support on OSX is an especially brutal form of hell, so it has got to be tested there too.

comment:27 in reply to: ↑ 26 Changed 10 months ago by fbissey

Replying to dimpase:

PS. I hear that openmp support on OSX is an especially brutal form of hell, so it has got to be tested there too.

If I remember correctly it is just turned off as soon as darwin is detected.

comment:28 Changed 10 months ago by dimpase

For one, if openmp is not installed then -fopenmp should not even be mentioned in the calls, but it does get mentioned.

comment:29 Changed 10 months ago by fbissey

We unfortunately deal with a meta build driven by a master makefile. No configuration is done apart possibly some flags passed to make.

comment:30 Changed 10 months ago by fbissey

The good stuff to look at configuration and what they do SuiteSparse_config/SuiteSparse_config.mk and to see what happen with the flags you want make ${my_flags} config I actually run it at the beginning of the build so it is logged. Setting CFOPENMP="" should kill openmp.

comment:31 Changed 10 months ago by dimpase

I'd rather see a way to use system's SuiteSparse? (as in #26286) rather than spending time on cutting custom tarballs...

comment:32 follow-up: Changed 10 months ago by fbissey

A lot of system split suitesparse so we would have to check the components individually. In any case I would not use spkg-src to make little tarballs. I would deal with it the same way gfortran and gcc are dealt with but on a bigger scale.

comment:33 in reply to: ↑ 24 Changed 10 months ago by dimpase

Replying to fbissey:

Yes it is unusual. And very strange. There is some control for openmp in the main makefile I think so it has to be turned off there (or autodetected properly there). But not finding stuff in /usr/local/lib is highly unusual - and it appears this is not your compiler's fault but ld's fault. Is it GNU ld, some kind of BSD ld or llvm's ld?

$ ld -v
LLD 6.0.1 (FreeBSD 335540-1200005) (compatible with GNU linkers)

comment:34 follow-up: Changed 10 months ago by fbissey

I believe this is llvm's ld then.

comment:35 in reply to: ↑ 34 ; follow-up: Changed 10 months ago by dimpase

Replying to fbissey:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

comment:36 in reply to: ↑ 35 ; follow-up: Changed 10 months ago by fbissey

Replying to dimpase:

Replying to fbissey:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

Definitely not called directly (I checked) the compiler is called to do the linking so -fopenmp should set all the flags properly. I am calling that a bug in linking driver of the compiler.

comment:37 in reply to: ↑ 36 Changed 10 months ago by dimpase

Replying to fbissey:

Replying to dimpase:

Replying to fbissey:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

Definitely not called directly (I checked) the compiler is called to do the linking so -fopenmp should set all the flags properly. I am calling that a bug in linking driver of the compiler.

I think I agree. It's ludicrous that LDFLAGS need to be set for openMP to work. And it appears to be well-known: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222858

comment:38 Changed 10 months ago by fbissey

Basically the -fopenmp switch doesn't work. Well this is half assed, it should either work or not work, but not partially work. Obviously it still add -lomp to the linker call.

comment:39 in reply to: ↑ 32 Changed 10 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

Replying to fbissey:

A lot of system split suitesparse so we would have to check the components individually. In any case I would not use spkg-src to make little tarballs. I would deal with it the same way gfortran and gcc are dealt with but on a bigger scale.

Checking components would be needed anyway, as it's installed as a bunch of separate libraries, whether they come from one blob or many.

Anyhow, send this one to the bots!

comment:40 Changed 10 months ago by fbissey

The separation would make it either to separate build and install. It would allow the "uninstall" work from Erik to work. Not all the libraries would be installed either.

comment:41 Changed 10 months ago by vbraun

  • Status changed from positive_review to needs_work

Fails on Cygwin

[suitesparse-5.3.0] ranlib libamd.a
[suitesparse-5.3.0] gcc -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -Wl,-rpath,/home/Admin/sage-buildbot-worker/sage_git/build/local/lib  -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -shared -Wl,-soname -Wl,libamd.so.2 -Wl,--no-undefined amd_i_aat.o amd_i_1.o amd_i_2.o amd_i_dump.o amd_i_postorder.o amd_i_defaults.o amd_i_post_tree.o amd_i_order.o amd_i_control.o amd_i_info.o amd_i_valid.o amd_i_preprocess.o amd_l_aat.o amd_l_1.o amd_l_2.o amd_l_dump.o amd_l_postorder.o amd_l_defaults.o amd_l_post_tree.o amd_l_order.o amd_l_control.o amd_l_info.o amd_l_valid.o amd_l_preprocess.o -o /home/Admin/sage-buildbot-worker/sage_git/build/local/lib/libamd.so.2.4.6 -lm -lsuitesparseconfig
[sagelib-8.5.beta5] running build_ext
[suitesparse-5.3.0] /usr/lib/gcc/x86_64-pc-cygwin/7.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lsuitesparseconfig
[suitesparse-5.3.0] collect2: error: ld returned 1 exit status
[suitesparse-5.3.0] make[6]: *** [Makefile:90: /home/Admin/sage-buildbot-worker/sage_git/build/local/lib/libamd.so.2.4.6] Error 1
[suitesparse-5.3.0] make[6]: Target 'install' not remade because of errors.

comment:42 Changed 10 months ago by fbissey

Hardly shocking. The build system being what it is. I think properly splitting the sub-package will help with that error. But it will take some time to do properly.

comment:43 Changed 8 months ago by embray

I didn't know the Cygwin buildbot was even working. Last I checked on it it was having problems with hanging on the doc build. I meant to look into it before vacation and then it just slipped off my radar. I'll look into it though.

comment:44 Changed 8 months ago by embray

  • Owner changed from (none) to embray
Note: See TracTickets for help on using tickets.