Coin-OR CBC: new-style SPKG

[ncohen]

This branch turns CBC into a new-style spkg.

SPKG: ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz (has to be renamed cbc-2.9.4.tar.gz and stored in SAGE_ROOT/upstream)

Nathann

[ncohen]

New commits:

​9e72fa1	trac #18126: copy of spkg-install and SPKG.txt from the old style spkg
​138570e	trac #18126: new checksum and package-version files
​97dd365	trac #18126: new spkg-src file to download and build the spkg

[dcoudert]

Are you sure it's IBM Coportation ?

I have the following issue, and I don't find it at that address.

confetti:sage dcoudert$ ./sage -i cbc
Found local metadata for cbc-2.9.4
Attempting to download package cbc-2.9.4
>>> Trying to download http://www.sagemath.org/packages/upstream/cbc/cbc-2.9.4.tar.bz2
[Traceback (most recent call last):
  File "<stdin>", line 35, in <module>
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 240, in retrieve
    fp = self.open(url, data)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 208, in open
    return getattr(self, name)(url)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 359, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 376, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "<stdin>", line 17, in http_error_default
IOError: [Errno 404] Not Found: '//www.sagemath.org/packages/upstream/cbc/cbc-2.9.4.tar.bz2'
Error: failed to download package cbc-2.9.4

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3bb47c9

trac #18126: License update

[ncohen]

Hellooooo,

Are you sure it's IBM Coportation ?

While updating that I noticed that the license had changed. Fixed.

I have the following issue, and I don't find it at that address.

This is because you do not have the new .tar.bz2 file. The spkg-src script (see the ticket's description) that this branch contains will download it for you.

Nathann

[dcoudert]

Hello,

still not working. Also, I don't see a cbc page at http://www.sagemath.org/packages/upstream/

Why not pointing directly to ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz ?

David.

[ncohen]

Hello,

still not working.

Could you give a more informative error message? What happens when you run spkg-src?

Also, I don't see a cbc page at http://www.sagemath.org/packages/upstream/

No problem here, that's because coin is an optional package. You don't see Nauty there either.

Why not pointing directly to ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz ?

Because it is not a .tar.bz2 file, and that's what I think we have to put in our upstream/ folder.

Nathann

[dcoudert]

Hello,

I should run spkg-src and not sage -i cbc ??? You should mention it in the ticket description.

confetti:sage dcoudert$ build/pkgs/cbc/spkg-src 
cat: package-version.txt: No such file or directory
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
       sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
--2015-04-06 14:08:36--  http://www.coin-or.org/download/source/Cbc/Cbc-.tgz
Résolution de www.coin-or.org (www.coin-or.org)… 130.127.215.8
Connexion à www.coin-or.org (www.coin-or.org)|130.127.215.8|:80… connecté.
requête HTTP transmise, en attente de la réponse… 404 Not Found
2015-04-06 14:08:39 erreur 404 : Not Found.

confetti:sage dcoudert$ ./sage -i cbc
Found local metadata for cbc-2.9.4
Attempting to download package cbc-2.9.4
>>> Trying to download http://www.sagemath.org/packages/upstream/cbc/cbc-2.9.4.tar.bz2
[Traceback (most recent call last):
  File "<stdin>", line 35, in <module>
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 240, in retrieve
    fp = self.open(url, data)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 208, in open
    return getattr(self, name)(url)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 359, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 376, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "<stdin>", line 17, in http_error_default
IOError: [Errno 404] Not Found: '//www.sagemath.org/packages/upstream/cbc/cbc-2.9.4.tar.bz2'
Error: failed to download package cbc-2.9.4

[ncohen]

Hello,

I should run spkg-src and not sage -i cbc ???

spkg-src downloads the upstream Cbc for you, transforms it a bit and stores it in upstream. Could you start this script while going in the right folder first?

The spkg-src scripts are just there to make the developer's life easier when a spkg gets udpated. You change the version number, start the script, and the .tar.bz2 file is built for you.

Once this is done, the command sage -i cbc will find the .tar.bz2 file.

Nathann

[ncohen]

Additionally, these spkg-src scripts are also useful to certify that the guy who uploads a .tar.bz2 file has not introduced any code in the original package. As you can see how the .tar.bz2 file is built, I cannot be suspected of introducing backdoors on everybody's computer.

Nathann

[dcoudert]

confetti:cbc dcoudert$ ./spkg-src 
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
       sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]
--2015-04-06 14:35:26--  http://www.coin-or.org/download/source/Cbc/Cbc-.tgz
Résolution de www.coin-or.org (www.coin-or.org)… 130.127.215.8
Connexion à www.coin-or.org (www.coin-or.org)|130.127.215.8|:80… connecté.
requête HTTP transmise, en attente de la réponse… 404 Not Found
2015-04-06 14:35:27 erreur 404 : Not Found.

confetti:cbc dcoudert$ cat package-version.txt | sed -re "s/^(.*)\\.p.+$/\\1/"
sed: illegal option -- r
usage: sed script [-Ealn] [-i extension] [file ...]
       sed [-Ealn] [-i extension] [-e script] ... [-f script_file] ... [file ...]

confetti:cbc dcoudert$ cat package-version.txt | sed -e "s/^(.*)\\.p.+$/\\1/"
sed: 1: "s/^(.*)\.p.+$/\1/
": \1 not defined in the RE

confetti:cbc dcoudert$ cat package-version.txt | sed -e "s/^(.*)\\.p.+$/$1/"
2.9.4

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​5c7e209

trac #18126: avoid -r in sed

[dcoudert]

Some progress

confetti:sage dcoudert$ ./sage -i cbc
Found local metadata for cbc-2.9.4
Attempting to download package cbc-2.9.4
>>> Trying to download http://www.sagemath.org/packages/upstream/cbc/cbc-2.9.4.tar.bz2
[Traceback (most recent call last):
  File "<stdin>", line 35, in <module>
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 240, in retrieve
    fp = self.open(url, data)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 208, in open
    return getattr(self, name)(url)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 359, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/Users/dcoudert/sage/local/lib/python/urllib.py", line 376, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "<stdin>", line 17, in http_error_default
IOError: [Errno 404] Not Found: '//www.sagemath.org/packages/upstream/cbc/cbc-2.9.4.tar.bz2'
Error: failed to download package cbc-2.9.4
confetti:sage dcoudert$          
confetti:sage dcoudert$ cd build/pkgs/cbc/
confetti:cbc dcoudert$ ./spkg-src 
--2015-04-06 14:48:46--  http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz
Résolution de www.coin-or.org (www.coin-or.org)… 130.127.215.8
Connexion à www.coin-or.org (www.coin-or.org)|130.127.215.8|:80… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 13761766 (13M) [application/x-gzip]
Sauvegarde en : « Cbc-2.9.4.tgz »

Cbc-2.9.4.tgz                                       100%[===================================================================================================================>]  13,12M  74,6KB/s   ds 2m 37s 

2015-04-06 14:51:24 (85,5 KB/s) — « Cbc-2.9.4.tgz » sauvegardé [13761766/13761766]

mv: rename Cbc-2.9.4.tar.bz2 to /upstream/cbc-2.9.4.tar.bz2: No such file or directory
confetti:cbc dcoudert$ ll
total 23032
-rw-r--r--  1 dcoudert  staff  11768072 31 mar 09:30 Cbc-2.9.4.tar.bz2
-rw-r--r--  1 dcoudert  staff       905  6 avr 14:04 SPKG.txt
-rw-r--r--  1 dcoudert  staff       128  6 avr 14:04 checksums.ini
-rw-r--r--  1 dcoudert  staff         6  6 avr 14:04 package-version.txt
-rwxr-xr-x  1 dcoudert  staff      1220  6 avr 14:04 spkg-install
-rwxr-xr-x  1 dcoudert  staff       429  6 avr 14:47 spkg-src

Also, version 2.9.4 is not listed on page http://www.sagemath.org/packages/optional/ Only 2.8.1.p0

[ncohen]

Hello,

This fails because your SAGE_ROOT variable is not set. You can:

1) Define it in your bash; or 2) Run the script from the console that you get after a sage -sh

Nathann

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​217fb18

trac #18126: Warning when SAGE_ROOT is not set

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​4e3830f

trac #18126: Warning when SAGE_ROOT is not set

[dcoudert]

You did a push while I was testing the stuff, so I don't know if it will change something.

After setting SAGE_ROOT in .bash_profile :

confetti:cbc dcoudert$ ./spkg-src 
--2015-04-06 15:07:53--  http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz
Résolution de www.coin-or.org (www.coin-or.org)… 130.127.215.8
Connexion à www.coin-or.org (www.coin-or.org)|130.127.215.8|:80… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 13761766 (13M) [application/x-gzip]
Sauvegarde en : « Cbc-2.9.4.tgz »

Cbc-2.9.4.tgz                                       100%[===================================================================================================================>]  13,12M  93,9KB/s   ds 2m 9s  

2015-04-06 15:10:05 (104 KB/s) — « Cbc-2.9.4.tgz » sauvegardé [13761766/13761766]

./spkg-src: line 17: sage: command not found
confetti:cbc dcoudert$ echo $SAGE_ROOT
/Users/dcoudert/sage/
confetti:cbc dcoudert$ sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath Version 6.6.rc2, Release Date: 2015-04-02                 │
│ Type "notebook()" for the browser-based notebook interface.        │
│ Type "help()" for help.                                            │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
sage: 
Exiting Sage (CPU time 0m0.02s, Wall time 0m5.00s).
confetti:cbc dcoudert$ cd $SAGE_ROOT
confetti:sage dcoudert$ sage -i cbc
Found local metadata for cbc-2.9.4
Found local sources at /Users/dcoudert/sage/upstream/cbc-2.9.4.tar.bz2
Checksum: 6afb943dcc3f60bff525456ad175cf19f6a2e28e vs 6afb943dcc3f60bff525456ad175cf19f6a2e28e
cbc-2.9.4
====================================================
Setting up build directory for cbc-2.9.4
...
...
Successfully installed cbc-2.9.4
Deleting temporary build directory
/Users/dcoudert/sage/local/var/tmp/sage/build/cbc-2.9.4
Finished installing cbc-2.9.4.spkg

Warning: it might be needed to update the Sage library before
installed packages work: you should run 'make' from $SAGE_ROOT
before running Sage.

So it is working this way, but it was not working when trying first sage -i cbc. Don't know why.

What should I do next?

[ncohen]

Hello !

You did a push while I was testing the stuff, so I don't know if it will change something.

I added a warning to say something more useful when SAGE_ROOT is not defined. This block of 4 lines can be found in most spkg-install scripts that Sage ships.

So it is working this way, but it was not working when trying first sage -i cbc. Don't know why.

New-style spkg work this way: ideally we have in upstream/ the copy of the source files, as given by the original authors of the software. Then, in build/pkgs we have some scripts to install it as it should in Sage.

As the sources were not available on your computer (and obviously not on Sage's server either, as the version we have here is more recent) the spkg-src script downloaded it from Coin's website and created the upstream/ archive. Then, with sage -i cbc you called the scripts from build/pkgs/, which used the new .tar.bz2 file.

What should I do next?

Well, now you apparently have Cbc on your computer. So you can solve LP with it, check that tests pass when you add --optional=sage,coin to sage -t.

Nathann

[tmonteil]

• The changes made on the source code should better be in a separate patch file rather than a sed command within the spkg-install script (so that one can easily see what changed from upstream in a uniform way). The associated comment should be part of the patch, see #17357
• I do not see the point of spkg-src : since there is no change made in the tarball, why not just keep the upstream Cbc-2.9.4.tgz file and rename it cbc-2.9.4.tar.gz without any modification and put the link ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz in the ticket description ?
• "SPKG Maintainers" field is no longer required, see ​https://groups.google.com/forum/?fromgroups#!searchin/sage-devel/spkg$20maintainers/sage-devel/-MZAvj8qxVk/EdbRFw0hGucJ ​http://thread.gmane.org/gmane.comp.mathematics.sage.devel/75324

[ncohen]

Hello,

• The changes made on the source code should better be in a separate patch file rather than a sed command within the spkg-install script (so that one can easily see what changed from upstream in a uniform way). The associated comment should be part of the patch, see #17357

The spkg-install file is a pure copy of what the spkg already contains.

• I do not see the point of spkg-src : since there is no change made in the tarball, why not just keep the upstream Cbc-2.9.4.tgz file and rename it cbc-2.9.4.tar.gz without any modification and put the link ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz in the ticket description ?

Because I believe that we need .tar.bz2 files in upstream. There is no other reason.

• "SPKG Maintainers" field is no longer required, see ​https://groups.google.com/forum/?fromgroups#!searchin/sage-devel/spkg$20maintainers/sage-devel/-MZAvj8qxVk/EdbRFw0hGucJ ​http://thread.gmane.org/gmane.comp.mathematics.sage.devel/75324

I don't mind. I will remove it in a second and set the patch to needs_review again, waiting for your answer about the first two points.

Nathann

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a372011

trac #18126: Remove the 'maintainers' section from SPKG.txt

[tmonteil]

Hi,

the third point was the minor one,

• concerning the first point, it is better to have our custom changes in a single well-defined place, you can have a look at : http://www.sagemath.org/doc/developer/packaging.html#patching-sources
• concerning the second point, tar.gz files are also accepted and spkg-src should be avoided as possible, you can have a look at : http://www.sagemath.org/doc/developer/packaging.html#modified-tarballs

Ciao, Thierry

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7a67d4d

trac #18126: Remove spkg-src

[ncohen]

Hello,

• concerning the first point, it is better to have our custom changes in a single well-defined place, you can have a look at : http://www.sagemath.org/doc/developer/packaging.html#patching-sources

In general applying patches is a good idea, but here we are patching an automatically generated file. I believe that it is safer to change it with a sed command (which just replaces some text with another one), as we would otherwise have to change the patch files (using the sed comand) whenever it changes.

• concerning the second point, tar.gz files are also accepted and spkg-src should be avoided as possible, you can have a look at : http://www.sagemath.org/doc/developer/packaging.html#modified-tarballs

I removed the script.

Nathann

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

[ncohen]

Having a .tar.gz file seems to break the install scipts. I removed the last commit and set the ticket's description to its original text.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7f8eda5

#18126 : use original .tar.gz source instead of custom .tar.bz2

[tmonteil]

Replying to ncohen:

Having a .tar.gz file seems to break the install scipts.

I think this is because you did not update checksums.ini.

The installation works fine, but then, when i do sage -b, i got :

...
building 'sage.numerical.backends.coin_backend' extension
Executing 1 command (using 1 thread)
error: [Errno 13] Permission denied

Do you know what could be wrong ?

[ncohen]

Yo !

I think this is because you did not update checksums.ini.

I am an idiot T_T

The installation works fine, but then, when i do sage -b, i got :

...
building 'sage.numerical.backends.coin_backend' extension
Executing 1 command (using 1 thread)
error: [Errno 13] Permission denied

Do you know what could be wrong ?

No idea. What is the line that raises that? Are you the user with writing permissions on SAGE_ROOT and its subdirectories?

Nathann

[tmonteil]

Replying to ncohen:

No idea. What is the line that raises that?

The full output of sage -b command is:

scons: `install' is up to date.
Updating Cython code....
Enabling Cython debugging support
Finished Cythonizing, time: 14.40 seconds.
Discovering Python source code....
Discovered Python source, time: 0.23 seconds.
Cleaning up stale installed files....
- cleaning /opt/sagemath/sage-6.3/local/lib/python2.7/site-packages
- cleaning /opt/sagemath/sage-6.3/local/lib/site-python
- cleaning /opt/sagemath/sage-6.3/src/build/lib.linux-i686-2.7
Finished cleaning, time: 0.29 seconds.
running install
running build
running build_py
running build_ext
building 'sage.numerical.backends.coin_backend' extension
Executing 1 command (using 1 thread)
error: [Errno 13] Permission denied

Note that sage -i cbc ended with Finished installing cbc-2.9.4.spkg. I have been out of memory during sd66, i hope it did not mess up something on my directory.

Are you the user with writing permissions on SAGE_ROOT and its subdirectories?

Yes, i have only one user on my computer :)

[fbissey]

Replying to tmonteil:

Yes, i have only one user on my computer :)

But that doesn't mean the folder belongs to you, there are plenty of system class user out there - root in particular. Have you done anything as root with your install?

[tmonteil]

Replying to fbissey:

Have you done anything as root with your install?

Not that i recall, but perhaps some chmod operation was not allowed when my disk was full. I double-checked that all files and subdirectories of SAGE_ROOT belong to my daily user (find ! -user 1000 leads to nothing). I removed all cbc-related code, and the problem is still there, so it is probably an unrelated issue, sorry for the noise, i will come back to the review once this is fixed.

[ncohen]

I set the ticket back to needs_review in the meantime. I have seen tickets go forgotten just because of their status :-P

Nathann

[vdelecroix]

The following worked well:

• downloading the tarball from ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz
• rename it to cbc-2.9.4.tar.gz and move it to $SAGE_ROOT/upstream
• do sage -i cbc
• do make

Could you update the ticket description?

Vincent

[vdelecroix]

Replying to ncohen:

Note that the installation with .tgz did not work for me and I had to rename it .tar.gz.

Vincent

[ncohen]

Note that the installation with .tgz did not work for me and I had to rename it .tar.gz.

I had not noticed. Could you make the modification so that I do not get it wrong again?

Nathann

[vdelecroix]

Running sage -t numerical/backends/coin_backend.pyx --optional Coin I got a lot of errors claiming that get_solver is not defined. Could you update the doctests in that ticket?

Vincent

[ncohen]

you must run then with --optional=sage,coin. Otherwise ONLY the lines with a flag get executed.

Nathann

[vdelecroix]

Much better ;-)

[jdemeyer]

Please remove this message

ATTENTION: YOU HAVE TO REBUILD THE SAGE LIBRARY TO GET THIS TO WORK

After building a package, there is already

Warning: it might be needed to update the Sage library before
installed packages work: you should run 'make' from $SAGE_ROOT
before running Sage.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a328985

trac #18126: Review

[vbraun]

Download link with the correct filename?

[ncohen]

Download link with the correct filename?

{{{wget ​http://www.coin-or.org/download/source/Cbc/Cbc-2.9.4.tgz -O cbc-2.9.4.tar.gz }}}

[vbraun]

Thanks, if you would also please integrate the renaming into the release management script?

[vbraun]

Thanks, if you would also please integrate the renaming into the release management script?

[ncohen]

Do you need a permanent url for the package with the right name?

Nathann

[vbraun]

yes, that is preferred

[ncohen]

What for? Don't you download it just once, to add it to Sage's mirrors?

Nathann

[vbraun]

upstream might also modify their tarball while we are waiting for sage-6.6

[ncohen]

We have the checksums, no worries.

Nathann

[vbraun]

I'm not testing optional package checksums fyi

[ncohen]

Would it be better for you if I re-wrote the spkg-src script that we had before? It would download the file and renames it for you.

Nathann

[ncohen]

Volker?

[vbraun]

How's that going to help? I'm not going to manually run a different script every time some tarball gets updated.

[ncohen]

How's that going to help? I'm not going to manually run a different script every time some tarball gets updated.

Well right now you have to download it manually [1]. They don't upload the tarballs on my server when they make a new release, and unfortunately you cannot download it directly yourself because the first letter is in upper case. Life is tough.

Nathann

[1] ​http://www.steinertriples.fr/ncohen/tmp/cbc-2.9.4.tar.gz