Upgrade to cvxopt 1.2.x

[dimpase]

This ticket is to upgrade to cvxopt 1.2.3

• Download page: ​https://pypi.org/project/cvxopt/#files
• Tarballs:
  • ​https://files.pythonhosted.org/packages/00/02/13657bcd12d93f53b64a19b68d204f43d5f3247c29287c9eaeefcfa58fb9/cvxopt-1.2.3.tar.gz
  • ​https://github.com/DrTimothyAldenDavis/SuiteSparse/archive/v5.6.0.tar.gz

[Somehow downloads as SuiteSparse-5.6.0.tar.gz in chromium] Since cvxopt 1.1.9, ​SuiteSparse is no longer included, thus it will have to be packaged separately.

Ticket #19687 was our previous upgrade, to cvxopt 1.1.8 (released 2015-09-22).

[dimpase]

suite-sparse license

[dimpase]

the 1st task would be to understand whether the Suite tarfile needs to be cut to exclude non-GPL compatible parts (see attachment)...

[fbissey]

I don't know about debian and other but we split suitesparse in components in Gentoo. I would be shocked if the other didn't do the same. It looks like upstream integrated a lot of the autoconfig patches that were floating around which makes life easier. What cvxopt needs:

• suitesparseconfig

*amd

*umfpack

*colamd

*cholmod

Indentation is for dependency. That's without optional or more complicated configuration - a minimal setup (although options using blas/lapack could be enabled without extra burden). All the above in mimimum configuration is LGPL 2.1. In non-mimimum configuration some switch to GPL-2.

[fbissey]

From setup.py - this can all be overridden, but it horrifies me

# Guess SUITESPARSE_LIB_DIR and SUITESPARSE_INC_DIR
if platform.platform().startswith("Darwin"):
    # macOS
    SUITESPARSE_LIB_DIR = '/usr/local/lib'
    SUITESPARSE_INC_DIR = '/usr/local/include'
else: 
    # default: Ubuntu/Debian
    SUITESPARSE_LIB_DIR = "/usr/lib/x86_64-linux-gnu"
    SUITESPARSE_INC_DIR = "/usr/include/suitesparse"

The only line that is valid is the default include directory for debian. The rest is accessible by default without defining anything. Basically it destroys stuff that would work out of the box without any values. This is relevant in that it means we will have to override things manually when it could work out of the box if left alone. That's not just suitesparse. gsl, glpk... Of course there are limitations of distutils that complicate things (an empty includedir statement generate a -I in the compilation line instead of nothing at all, the compiler then gets confused. The best default is probably '.' if you have to put something).

[fbissey]

Good news, cvxopt-1.1.9 is in gentoo now and no doctests were harmed by the upgrade in sage-on-gentoo. The bad news is that writing the spkgs for this is not high on my TODO list.

[slelievre]

Sage-on-Gentoo just upgraded to cvxopt 1.2.1.

See ​https://github.com/cschwan/sage-on-gentoo/issues/524

[dimpase]

It would be great if Gentoo's suite-sparse can be lifted to become a Sage package.

For starters I should get sage-on-gentoo running on my Gentoo laptop. How do I start with it?

[fbissey]

Replying to dimpase:

It would be great if Gentoo's suite-sparse can be lifted to become a Sage package.

For starters I should get sage-on-gentoo running on my Gentoo laptop. How do I start with it?

Well we are a bit old school but the instructions of the README at ​https://github.com/cschwan/sage-on-gentoo are still working. One day I'll update them to something a little bit more modern. Note that you also need the science overlay. I won't finish sage 8.4 tongiht I am afraid. Too much to do, it will be ready in 12 hours.

[Konrad127123]

Note: cvxopt 1.2.2 was released a few days ago.

[slelievre]

Updating ticket description following the release of cvxopt 1.2.2.

[dimpase]

Replying to fbissey:

Well we are a bit old school but the instructions of the README at ​https://github.com/cschwan/sage-on-gentoo are still working.

A somewhat cryptic error:

!!! The ebuild selected to satisfy "sage" has unmet requirements.
- sci-mathematics/sage-8.4::sage-on-gentoo USE="X doc-html-bin -bliss -debug -doc-html -doc-pdf -doc-pdf-bin -jmol -latex -modular_decomposition -sagenb -testsuite" ABI_X86="(64)" L10N="-ca -de -en -es -fr -hu -it -ja -pt -ru -tr" PYTHON_TARGETS="python2_7 python3_6"

  The following REQUIRED_USE flag constraints are unsatisfied:
    doc-html-bin? ( l10n_en )

  The above constraints are a subset of the following complete expression:
    doc-html? ( l10n_en sagenb ) doc-pdf? ( sagenb ) testsuite? ( any-of ( doc-html doc-html-bin ) ) doc-html-bin? ( !doc-html !doc-pdf l10n_en ) doc-pdf-bin? ( !doc-html !doc-pdf l10n_en ) any-of ( python_targets_python2_7 python_targets_python3_6 )

no idea what's wrong. Changing doc-html-bin to -doc-html-bin allows the build to go forward.

[fbissey]

That's part of the possible useflags. Basically I allow you to choose which language you want to build but because the english version is the only complete one, if you elect to have documentation, the english one is compulsory. Put

L10N="en"

in make.conf and you'll get pre-built english documentation installed.

[fbissey]

I am going to make an attempt at a "reduced" suitesparse spkg. Reduced because I will not make default packages (which aren't used by anything else) that use cmake. Note that suitesparse will move to cmake for the whole thing in the near future.

The fun bit is that suitesparse is effectively a small mini distribution itself (but most of the upstreams are now totally absorbed by the project). It doesn't currently work very nicely with building phases unless we break down the various components. It does install directly on the file system and because it is like a mini distribution, trying to stop it will break the build system (where are my libraries and my include files).

Once it is completely cmake-ified it may actually behave better in that respect.

[fbissey]

SPKG.txt for suitesparse needs a bit of polish but this is a good start for anyone who wants to tests this. This is open for reviews even though I am expecting a bit more work needs to be done.

---

New commits:

​94517d2	basic suitsparse spkg
​903741f	initial update to cvxopt 1.2.2
​093e5f3	Merge branch 'develop' into cvxopt-1.22
​32e028e	add suitesparse to cvxopt dependencies

[Konrad127123]

I'm getting an Invalid checksum error for cvxopt-1.2.2.tar.gz.

cvxopt/checksums.ini says:

tarball=cvxopt-VERSION.tar.gz
sha1=725ce85fe122cd0fcda9d1b1a45ae43adf6fea2c
md5=ef9ab5f939380e9488e57b113103d96a
cksum=4099798435

but I get:

$ sha1sum ../../upstream/cvxopt-1.2.2.tar.gz 
43113a138e5e8f3ba42dc53ccb8c3c7501d5576b  ../../upstream/cvxopt-1.2.2.tar.gz
$ md5sum ../../upstream/cvxopt-1.2.2.tar.gz 
fc2546fdba2faa6c4996e9b40fce5269  ../../upstream/cvxopt-1.2.2.tar.gz
$ cksum ../../upstream/cvxopt-1.2.2.tar.gz
3288957477 1877253 ../../upstream/cvxopt-1.2.2.tar.gz

[fbissey]

My bad I used the wrong tarball (github instead of pypi) when creating the checksum. I'll fix it shortly.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​21e7280

fix cvxopt checksum

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​87f0048

Flesh out SPKG.txt

[fbissey]

I would be OK for things to ship as they are now but I am open to suggestions. An alternative is to split SuiteSparse into components but that's more work.

[Konrad127123]

Building with SAGE_CHECK="yes" still fails when running headless (#24657).

[fbissey]

I see, I was not aware of this issue. I see that matplotlib is involved, probably in the same way some doctest could be failing on debian. That will require some code inspection.

[dimpase]

Without openmp, or with openmp installed in /usr/local/, I get I get

[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)

Is openmp a pre-req?

One way or another, the installation ought to be smarter as far as the location of openmp is concerned.

[fbissey]

Yes it should be smarter. Now can you give me more info? Is this OS X? which sub-package of suitesparse? In short a longer log please.

[dimpase]

This is FreeBSD 12.0. With openmp installed (in /usr/local, which is "standard" location on such a system), everything works (passes relevant Sage tests too) if I export LDFLAGS="$LDFLAGS -L/usr/local/lib" before installation. If LDFLAGS are not passed, it breaks at CHOLMOD as follows:

...
[suitesparse-5.3.0] a - cholmod_l_nesdis.o
[suitesparse-5.3.0] a - cholmod_l_camd.o
[suitesparse-5.3.0] cc -L/usr/home/dimpase/Sage/sagemath/local/lib -Wl,-rpath,/usr/home/dimpase/Sage/sagemath/local/lib  -L/usr/home/dimpase/Sage/sagemath/local/lib -L/usr/home/dimpase/Sage/sagemath/local/lib -L/usr/home/dimpase/Sage/sagemath/local/lib -shared -Wl,-soname -Wl,libcholmod.so.3 -Wl,--no-undefined -fopenmp cholmod_aat.o cholmod_add.o cholmod_band.o cholmod_change_factor.o cholmod_common.o cholmod_complex.o cholmod_copy.o cholmod_dense.o cholmod_error.o cholmod_factor.o cholmod_memory.o cholmod_sparse.o cholmod_transpose.o cholmod_triplet.o cholmod_version.o cholmod_check.o cholmod_read.o cholmod_write.o cholmod_amd.o cholmod_analyze.o cholmod_colamd.o cholmod_etree.o cholmod_factorize.o cholmod_postorder.o cholmod_rcond.o cholmod_resymbol.o cholmod_rowcolcounts.o cholmod_rowfac.o cholmod_solve.o cholmod_spsolve.o cholmod_drop.o cholmod_horzcat.o cholmod_norm.o cholmod_scale.o cholmod_sdmult.o cholmod_ssmult.o cholmod_submatrix.o cholmod_vertcat.o cholmod_symmetry.o cholmod_rowadd.o cholmod_rowdel.o cholmod_updown.o cholmod_super_numeric.o cholmod_super_solve.o cholmod_super_symbolic.o cholmod_ccolamd.o cholmod_csymamd.o cholmod_metis.o cholmod_nesdis.o cholmod_camd.o cholmod_l_aat.o cholmod_l_add.o cholmod_l_band.o cholmod_l_change_factor.o cholmod_l_common.o cholmod_l_complex.o cholmod_l_copy.o cholmod_l_dense.o cholmod_l_error.o cholmod_l_factor.o cholmod_l_memory.o cholmod_l_sparse.o cholmod_l_transpose.o cholmod_l_triplet.o cholmod_l_version.o cholmod_l_check.o cholmod_l_read.o cholmod_l_write.o cholmod_l_amd.o cholmod_l_analyze.o cholmod_l_colamd.o cholmod_l_etree.o cholmod_l_factorize.o cholmod_l_postorder.o cholmod_l_rcond.o cholmod_l_resymbol.o cholmod_l_rowcolcounts.o cholmod_l_rowfac.o cholmod_l_solve.o cholmod_l_spsolve.o cholmod_l_drop.o cholmod_l_horzcat.o cholmod_l_norm.o cholmod_l_scale.o cholmod_l_sdmult.o cholmod_l_ssmult.o cholmod_l_submatrix.o cholmod_l_vertcat.o cholmod_l_symmetry.o cholmod_l_rowadd.o cholmod_l_rowdel.o cholmod_l_updown.o cholmod_l_super_numeric.o cholmod_l_super_solve.o cholmod_l_super_symbolic.o cholmod_l_ccolamd.o cholmod_l_csymamd.o cholmod_l_metis.o cholmod_l_nesdis.o cholmod_l_camd.o -o /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12 -lm -lamd -lcolamd -lsuitesparseconfig -lccolamd -lcamd -lopenblas  -lopenblas 
[suitesparse-5.3.0] ranlib libcholmod.a
[suitesparse-5.3.0] /usr/bin/ld: error: unable to find library -lomp
[suitesparse-5.3.0] cc: error: linker command failed with exit code 1 (use -v to see invocation)
[suitesparse-5.3.0] gmake[4]: *** [Makefile:544: /usr/home/dimpase/Sage/sagemath/local/lib/libcholmod.so.3.0.12] Error 1
[suitesparse-5.3.0] gmake[4]: Leaving directory '/usr/home/dimpase/Sage/sagemath/local/var/tmp/sage/build/suitesparse-5.3.0/src/CHOLMOD/Lib'

[fbissey]

Yes it is unusual. And very strange. There is some control for openmp in the main makefile I think so it has to be turned off there (or autodetected properly there). But not finding stuff in /usr/local/lib is highly unusual - and it appears this is not your compiler's fault but ld's fault. Is it GNU ld, some kind of BSD ld or llvm's ld?

[fbissey]

Is there urgency to have this in 8.5? I have been thinking that I have a longer time frame I may want to try splitting suitesparse into just the components needed for cvxopt.

[dimpase]

PS. I hear that openmp support on OSX is an especially brutal form of hell, so it has got to be tested there too.

[fbissey]

Replying to dimpase:

PS. I hear that openmp support on OSX is an especially brutal form of hell, so it has got to be tested there too.

If I remember correctly it is just turned off as soon as darwin is detected.

[dimpase]

For one, if openmp is not installed then -fopenmp should not even be mentioned in the calls, but it does get mentioned.

[fbissey]

We unfortunately deal with a meta build driven by a master makefile. No configuration is done apart possibly some flags passed to make.

[fbissey]

The good stuff to look at configuration and what they do SuiteSparse_config/SuiteSparse_config.mk and to see what happen with the flags you want make ${my_flags} config I actually run it at the beginning of the build so it is logged. Setting CFOPENMP="" should kill openmp.

[dimpase]

I'd rather see a way to use system's SuiteSparse? (as in #26286) rather than spending time on cutting custom tarballs...

[fbissey]

A lot of system split suitesparse so we would have to check the components individually. In any case I would not use spkg-src to make little tarballs. I would deal with it the same way gfortran and gcc are dealt with but on a bigger scale.

[dimpase]

Replying to fbissey:

Yes it is unusual. And very strange. There is some control for openmp in the main makefile I think so it has to be turned off there (or autodetected properly there). But not finding stuff in /usr/local/lib is highly unusual - and it appears this is not your compiler's fault but ld's fault. Is it GNU ld, some kind of BSD ld or llvm's ld?

$ ld -v
LLD 6.0.1 (FreeBSD 335540-1200005) (compatible with GNU linkers)

[fbissey]

I believe this is llvm's ld then.

[dimpase]

Replying to fbissey:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

[fbissey]

Replying to dimpase:

Replying to fbissey:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

Definitely not called directly (I checked) the compiler is called to do the linking so -fopenmp should set all the flags properly. I am calling that a bug in linking driver of the compiler.

[dimpase]

Replying to fbissey:

Replying to dimpase:

Replying to fbissey:

I believe this is llvm's ld then.

yep. If it is called directly, it does not know where to look for libraries, so one much pass -L/lib -L/usr/lib -L etc to it.

Definitely not called directly (I checked) the compiler is called to do the linking so -fopenmp should set all the flags properly. I am calling that a bug in linking driver of the compiler.

I think I agree. It's ludicrous that LDFLAGS need to be set for openMP to work. And it appears to be well-known: ​https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222858

[fbissey]

Basically the -fopenmp switch doesn't work. Well this is half assed, it should either work or not work, but not partially work. Obviously it still add -lomp to the linker call.

[dimpase]

Replying to fbissey:

A lot of system split suitesparse so we would have to check the components individually. In any case I would not use spkg-src to make little tarballs. I would deal with it the same way gfortran and gcc are dealt with but on a bigger scale.

Checking components would be needed anyway, as it's installed as a bunch of separate libraries, whether they come from one blob or many.

Anyhow, send this one to the bots!

[fbissey]

The separation would make it either to separate build and install. It would allow the "uninstall" work from Erik to work. Not all the libraries would be installed either.

[vbraun]

Fails on Cygwin

[suitesparse-5.3.0] ranlib libamd.a
[suitesparse-5.3.0] gcc -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -Wl,-rpath,/home/Admin/sage-buildbot-worker/sage_git/build/local/lib  -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -L/home/Admin/sage-buildbot-worker/sage_git/build/local/lib -shared -Wl,-soname -Wl,libamd.so.2 -Wl,--no-undefined amd_i_aat.o amd_i_1.o amd_i_2.o amd_i_dump.o amd_i_postorder.o amd_i_defaults.o amd_i_post_tree.o amd_i_order.o amd_i_control.o amd_i_info.o amd_i_valid.o amd_i_preprocess.o amd_l_aat.o amd_l_1.o amd_l_2.o amd_l_dump.o amd_l_postorder.o amd_l_defaults.o amd_l_post_tree.o amd_l_order.o amd_l_control.o amd_l_info.o amd_l_valid.o amd_l_preprocess.o -o /home/Admin/sage-buildbot-worker/sage_git/build/local/lib/libamd.so.2.4.6 -lm -lsuitesparseconfig
[sagelib-8.5.beta5] running build_ext
[suitesparse-5.3.0] /usr/lib/gcc/x86_64-pc-cygwin/7.3.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lsuitesparseconfig
[suitesparse-5.3.0] collect2: error: ld returned 1 exit status
[suitesparse-5.3.0] make[6]: *** [Makefile:90: /home/Admin/sage-buildbot-worker/sage_git/build/local/lib/libamd.so.2.4.6] Error 1
[suitesparse-5.3.0] make[6]: Target 'install' not remade because of errors.

[fbissey]

Hardly shocking. The build system being what it is. I think properly splitting the sub-package will help with that error. But it will take some time to do properly.

[embray]

I didn't know the Cygwin buildbot was even working. Last I checked on it it was having problems with hanging on the doc build. I meant to look into it before vacation and then it just slipped off my radar. I'll look into it though.

[jhpalmieri]

In the course of fixing this, I request that the testsuite be fixed, too. Newer versions of cvxopt use nose, which is a standard Sage package, so testing should be easy to implement. (The current versions have a more ad hoc system which is currently broken, and so is being removed at #28730.)

[fbissey]

I'll be trying to blow new life in this ticket soon. I may use a split suitesparse based on my gentoo work.

[dimpase]

cvxopt is meanwhile version 1.2.3

[fbissey]

Basic bump to latest versions without any splitting of suitesparse. In the event of a split with need the following components

• suitesparseconfig
• amd
• colamd
• cholmod
• umfpack

The order is indicative or the compilation order [OK amd and colamd do not depend on each other].

This doesn't include a new spkg-check. I want to defer that to a positive review of this part on cygwin at least.

---

New commits:

​39316e0	Merge branch 'develop' into cvxopt-1.2.x
​a946e6e	Update to cvxopt-1.2.3 and suitesparse 5.6.0

[dimpase]

I think we should proceed with it without splitting the suitesparse. Well, 55Mb, so what. If anyone likes to make it smaller, they should volunteer...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e23d5c5

adding cvxopt testsuite with nose

[fbissey]

Added spkg-check after testing that it is working. Shamelessly copied it from the networkx one with appropriate adjustments.

[jhpalmieri]

Replying to dimpase:

I think we should proceed with it without splitting the suitesparse. Well, 55Mb, so what. If anyone likes to make it smaller, they should volunteer...

I think that the full suitesparse requires cmake, and I'd rather not have to make cmake a standard package.

[fbissey]

Replying to jhpalmieri:

Replying to dimpase:

I think we should proceed with it without splitting the suitesparse. Well, 55Mb, so what. If anyone likes to make it smaller, they should volunteer...

I think that the full suitesparse requires cmake, and I'd rather not have to make cmake a standard package.

I specifically disabled the bits requiring cmake. The packages requiring cmake are not used by any other of the packages. It somewhat feels like someone is packaging some pet packages along the more widely used stuff.

[jhpalmieri]

Yes, it's fine to include the full suitesparse, as long as we don't build the full suitesparse.

[fbissey]

I guess I could go "redux" and only install what we need. Note that some bits of suitesparse are also bundled (as in vendored) in glpk. And because glpk is autotooled, removing the bundled bits would require patching configure and generating a sage specific tarball.

[fbissey]

And now I remember we already unvendor zlib from glpk. May as well unvendor amd and colamd in a later ticket if we ship suitesparse (even in redux form since those are also needed by cvxopt).

[jhpalmieri]

By the way, with OS X 10.14.6, the cvxopt test suite passes, and Sage passes its tests.

[fbissey]

Any volunteer to positive review this, or is there anything you wish me to change?

[dimpase]

ok. not tested on cygwin, but it should not stop it (I think cvxopt is well tested on windows)

[embray]

Replying to dimpase:

ok. not tested on cygwin, but it should not stop it (I think cvxopt is well tested on windows)

It should have--if you needed me to I would have done it, but I don't have time to check every issue so you need to ping me directly if it seems I'm not responding.