Opened 9 years ago

Closed 9 years ago

#13160 closed enhancement (fixed)

upgrade cvxopt to 1.1.5

Reported by: fbissey Owned by: tbd
Priority: major Milestone: sage-5.3
Component: packages: standard Keywords:
Cc: Merged in: sage-5.3.beta1
Authors: François Bissey Reviewers: Volker Braun
Report Upstream: N/A Work issues:
Branch: Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by vbraun)

Upgrading cvxopt to 1.1.5 we can get rid of a patch to correct a wrong behavior that is obvious on OS X. Also I want to simplify the setup.py patch which give rise to overlinking (and in fact potential linking of multiple cblas libraries - yuk!). I am also attempting to install the documentation.

Updated spkg at:

Change History (34)

comment:1 Changed 9 years ago by fbissey

I have a spkg almost ready to upload. Changes are uncommitted yet. There are some kinks to check in the installation of the documentation.

comment:2 Changed 9 years ago by fbissey

  • Description modified (diff)

Experimental spkg now posted and linked in the description. The changes in the spkg have not been committed yet so I can still fiddle with it as necessary.

comment:3 Changed 9 years ago by dimpase

  • Dependencies set to #10508
  • Status changed from new to needs_info

Looks good on OSX 10.6.8 and Sage 5.2.beta1 with #10508 applied. I tried both lapack/blas, the ones from Apple, and the ones from Atlas 3.10 from #10508, with SAGE_CHECK on.

By the way, shouldn't #10508 be a dependence of this ticket? (I added it here, tentatively.)

Also, I wonder if there is anything to report upstream here.

comment:4 follow-up: Changed 9 years ago by vbraun

  • Dependencies #10508 deleted

#10508 doesn't change the layout of the ATLAS libraries, the current atlas-3.8.4 already provides the cblas library. If the cvxopt spkg were to link to the correct libraries only, it would work with both atlas-3.8.4 and atlas-3.10.0 spkgs.

comment:5 in reply to: ↑ 4 Changed 9 years ago by dimpase

Replying to vbraun:

#10508 doesn't change the layout of the ATLAS libraries, the current atlas-3.8.4 already provides the cblas library. If the cvxopt spkg were to link to the correct libraries only, it would work with both atlas-3.8.4 and atlas-3.10.0 spkgs.

When you closed #10509 you said it is fixed by atlas-3.10.0 spkg. And #10508 is the one the provides atlas-3.10.0 spkg.

comment:6 Changed 9 years ago by fbissey

The only thing really to check in this ticket is the installation of the documentation. Technically it needs sphinx to build but because up to date build documentation is shipped nothing is done. So the question is whether to require sphinx or just copy the already built documentation. Once I make an informed decision on that I'll finalize the spkg.

Last edited 9 years ago by fbissey (previous) (diff)

comment:7 follow-up: Changed 9 years ago by fbissey

OK I pushed a new spkg on google code it install the already built documentation and has a section commented out on how to build it in spkg-install.

comment:8 Changed 9 years ago by fbissey

  • Status changed from needs_info to needs_review

comment:9 in reply to: ↑ 7 ; follow-ups: Changed 9 years ago by dimpase

Replying to fbissey:

OK I pushed a new spkg on google code it install the already built documentation and has a section commented out on how to build it in spkg-install.

Shouldn't it install docs by default? (Currently, you need to set SAGE_SPKG_INSTALL_DOCS)

Also, I don't understand what you mean by "dependence" on sphinx. Isn't sphinx included in Sage?

comment:10 in reply to: ↑ 9 ; follow-up: Changed 9 years ago by fbissey

Replying to dimpase:

Replying to fbissey:

OK I pushed a new spkg on google code it install the already built documentation and has a section commented out on how to build it in spkg-install.

Shouldn't it install docs by default? (Currently, you need to set SAGE_SPKG_INSTALL_DOCS)

Also, I don't understand what you mean by "dependence" on sphinx. Isn't sphinx included in Sage?

As far as I know, you only install documentation for spkg if that variable is set. I may be wrong.

Yes sphinx is shipped with sage but if this spkg uses sphinx it has to be listed as one of its dependencies in spkg/standard/deps. In this particular case it is probably ok because cvxopt can be build after sage I think. But in the alternative there are problems as far as I remember from a previous upgrade of numpy way back.

comment:11 in reply to: ↑ 10 Changed 9 years ago by dimpase

Replying to fbissey:

OK, I asked on sage-devel

comment:12 in reply to: ↑ 9 Changed 9 years ago by kcrisman

Replying to dimpase:

Replying to fbissey:

OK I pushed a new spkg on google code it install the already built documentation and has a section commented out on how to build it in spkg-install.

Shouldn't it install docs by default? (Currently, you need to set SAGE_SPKG_INSTALL_DOCS)

?? #11197 has sort of delayed most packages actually implementing #10823. I believe that we currently don't want docs installed by default outside of wherever the spkg lives. Or maybe I'm misunderstanding the point of this discussion, my apologies if that's so.

comment:13 follow-up: Changed 9 years ago by fbissey

I didn't remember how the discussion had ended. Probably because I wasn't on the follow up ticket where things were discussed. Should I remove the section about documentation until one day we sort this out?

We should note that some packages already put their doc in $SAGE_LOCAL/share/doc:

frb15@p2n14-c /hpc/scratch/frb15/sandbox/sage-5.1.beta5/local/share/doc :ll   
total 1792
drwxr-xr-x 2 frb15 z001 131072 Jun 29 13:08 NTL
drwxr-xr-x 5 frb15 z001 131072 Jun 29 15:05 ipython
drwxr-xr-x 3 frb15 z001 131072 Jun 29 13:13 mpfr
drwxr-xr-x 3 frb15 z001 131072 Jun 29 15:12 networkx-1.2
drwxr-xr-x 3 frb15 z001 131072 Jun 29 13:53 ppl
drwxr-xr-x 3 frb15 z001 131072 Jun 29 13:53 pwl
drwxr-xr-x 2 frb15 z001 131072 Jun 29 15:49 sagetex

I am not even sure those obey SAGE_SPKG_INSTALL_DOCS and some are html. The cvxopt spkg linked in this ticket doesn't need sphinx because the doc is prebuilt in the tarball.

comment:14 in reply to: ↑ 13 Changed 9 years ago by dimpase

Replying to fbissey:

I didn't remember how the discussion had ended. Probably because I wasn't on the follow up ticket where things were discussed. Should I remove the section about documentation until one day we sort this out?

no, why, documentation is great to have. But why not remove the pre-built one and make it using sphynx? (the latter will need a trivial change in SAGE_ROOT/spkg/standard/deps).

comment:15 Changed 9 years ago by fbissey

Can do that but I don't think it will be much different from the one shipped. The first thing that sphinx does when it is run is check whether what is already built is up to date. Rebuilding it again is just a waste of cpu cycles.

I don't really care about it that much myself. So if people are quite happy about rebuilding the doc I'll just remove the pre build from the spkg to save space and produce the other needed patch.

comment:16 follow-up: Changed 9 years ago by dimpase

  • Status changed from needs_review to needs_work

There are leftovers of debugging in src/src/C/blas.c, starting from line 295, which reads

    printf("scal\n");

This is very strange.

EDIT: I checked the upstream, and they don't have these printfs...

Last edited 9 years ago by dimpase (previous) (diff)

comment:17 in reply to: ↑ 16 Changed 9 years ago by dimpase

Replying to dimpase:

There are leftovers of debugging in src/src/C/blas.c, starting from line 295, which reads

    printf("scal\n");

This is very strange.

EDIT: I checked the upstream, and they don't have these printfs...

Moreover, the upstream files base.c, blas.c, and sparse.c in src/src/C have earlier dates (and different sizes) than the ones in spkg. And replacing the spkg files by upstream files makes the package seemingly work on OSX10.6.8. I didn't test much, but the file src/examples/doc/chap10/l1svc.py which gives me crashes with the spkg, works OK with the upstream replacement.

comment:18 Changed 9 years ago by fbissey

Sorry wrong spkg I was tracking something on power7 I will correct the source today - hopefully.

comment:19 Changed 9 years ago by fbissey

  • Status changed from needs_work to needs_review

Corrected the source. Sorry for the inconvenience.

comment:20 Changed 9 years ago by vbraun

  • Description modified (diff)

On Fedora 17 x86_64:

building 'gsl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/C
gcc -fno-strict-aliasing -fwrapv -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/vbraun/opt/sage-atlas/sage-5.2/local/include/gsl -I/home/vbraun/opt/sage-atlas/sage-5.2/local/include/python2.7 -c C/gsl.c -o build/temp.linux-x86_64-2.7/C/gsl.o
C/gsl.c:28:25: fatal error: gsl/gsl_rng.h: No such file or directory
compilation terminated.

The header is in Sage but the include path is wrong:

[vbraun@laptop sage-5.2]$ find | grep gsl_rng
./local/include/gsl/gsl_rng.h

The fix is to set If I edit GSL_INC_DIR = SAGE_INCLUDE in setup.py, and not SAGE_INCLUDE/gsl.

I've updated the spkg (see description, only change is the GSL_INC_DIR) and it builds for me now.

comment:21 Changed 9 years ago by fbissey

OK you beat me to it. I was going to post about it. I am guessing the problem is not visible if gsl is installed system wide.

comment:22 Changed 9 years ago by vbraun

Just saw that we discovered the same bug within ~5mins. Coincidence? ;-)

comment:23 Changed 9 years ago by fbissey

John was the pointer for me really. Didn't notice it before - all the system I tested it with had gsl as standard. I need a box with nothing interesting on it really.

comment:24 Changed 9 years ago by jhpalmieri

On hawk, this builds if SAGE_ATLAS_LIB points to the system ATLAS, but not if I use the ATLAS spkg from #10508. Log here.

comment:25 Changed 9 years ago by fbissey

shouldn't be -lblas but -lf77blas I guess.

comment:26 Changed 9 years ago by fbissey

+if os.environ['UNAME'] == 'CYGWIN':
+  BLAS_LIB =['blas', 'gfortran']
+elif os.environ['UNAME'] == 'Darwin':
+  BLAS_LIB =['blas','gfortran','m']
+else:
+  BLAS_LIB =['blas','cblas','gfortran','atlas','m']
+

should really be

+if os.environ['UNAME'] == 'CYGWIN':
+  BLAS_LIB =['blas', 'gfortran']
+elif os.environ['UNAME'] == 'Darwin':
+  BLAS_LIB =['blas','gfortran','m']
+else:
+  BLAS_LIB =['f77blas','cblas','gfortran','atlas','m']
+

I think it may have escaped because we had libblas before #10508

comment:27 Changed 9 years ago by vbraun

I agree and just replaced the spkg (new md5sum = b155f7e4624c2869b9a0e5aa21803435)

Builds on Fedora 17

comment:28 Changed 9 years ago by jhpalmieri

Builds and passes self-tests on hawk, both with the system ATLAS and the ATLAS spkg from #10508.

comment:29 Changed 9 years ago by vbraun

  • Authors set to François Bissey
  • Reviewers set to Volker Braun
  • Status changed from needs_review to positive_review

Looks good to me!

comment:30 Changed 9 years ago by fbissey

Thanks Volker for the last mile. I also just thought that we will need a follow up ticket if we allow people to use ATLAS on OS X.

comment:31 follow-up: Changed 9 years ago by jhpalmieri

So on OS X, if people use the ATLAS spkg, then is this using a mix of Sage's ATLAS and the system's blas? (I'm in the process of building on OS X, including ATLAS from #10508, and cvxopt has built and passes its self-tests. The build hasn't completed, so Sage's test suite hasn't run yet.)

comment:32 in reply to: ↑ 31 Changed 9 years ago by dimpase

Replying to jhpalmieri:

So on OS X, if people use the ATLAS spkg, then is this using a mix of Sage's ATLAS and the system's blas? (I'm in the process of building on OS X, including ATLAS from #10508, and cvxopt has built and passes its self-tests. The build hasn't completed, so Sage's test suite hasn't run yet.)

I hope so. It would be a big speed digression to use GSL.

comment:33 Changed 9 years ago by fbissey

From what I put in the spkg cvxopt will use the OS X accelerate framework instead of ATLAS. You should note that it is not the only spkg for which it will be the case. linbox will do that as well for example. iml also comes to mind as well and numpy and scipy should be checked. From the sage shell can you check the output of

otool -L local/lib/python/site-packages/cvxopt/blas.so

It should tell you against which library it was linked in some form.

Last edited 9 years ago by fbissey (previous) (diff)

comment:34 Changed 9 years ago by jdemeyer

  • Merged in set to sage-5.3.beta1
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.