Add rules for installing packages with pip

[jdemeyer]

With the infrastructure of #19101, we can make it such that

sage -i pyopenssl

actually runs pip.

All the following optional packages need to be removed from the server as they are now supported by the above command:

• beautifulsoup
• biopython
• brian
• guppy
• mercurial
• mpi4py
• nibabel
• pybtex
• pyflakes
• pyopenssl
• sqlalchemy
• trac

follow-up: #19213, #19220

[jdemeyer]

New commits:

​93ec09b	Change sage -p to always install a package
​c6ddd0e	Better help
​f3d7ef3	Remove -f option to sage-spkg when running sage -i
​752e269	Use PKG-clean target to implement sage -f PKG
​1812e4f	Merge tag '6.9.beta6' into t/19119/ticket/19119
​adef308	Add rules for installing packages with pip

[kcrisman]

Can we do that for all packages that are installable that way (brian, biopython, etc.) then? (I tried to look at the commit to see if it did that but my internet is very flaky until I get to work.)

[jdemeyer]

Replying to kcrisman:

Can we do that for all packages that are installable that way?

The current branch deals with

beautifulsoup
biopython
brian
guppy
mpi4py
nibabel
pybtex
pyflakes
pyopenssl
trac

[kcrisman]

Thanks - I actually saw that just before my internet cut out on the train :-( so I couldn't edit my post - but I appreciate the quick followup :-) and good thinking to do this, of course, we should have long ago.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​fdba3ee

Add mercurial and sqlalchemy (ex-standard packages)

[kcrisman]

(You know, this could be very helpful for disentangling the many packages in sagenb.)

[jdemeyer]

Replying to kcrisman:

(You know, this could be very helpful for disentangling the many packages in sagenb.)

Actually not, because these special pip rules are not standard packages.

[vdelecroix]

Hi Jeroen

Do you consider your solution as temporary? I am afraid that it would become an open door to have thousands of packages in piprules. Shouldn't we reorient users toward sage -pip install X?

Vincent

[tscrim]

Could this also be modified to work for nzmath and pyx?

[kcrisman]

Do you consider your solution as temporary? I am afraid that it would become an open door to have thousands of packages in piprules.

That could be regarded as a feature. After all, we'd still want to have some test with respect to doctests or something. If people just want to add a pip package, whatever. If they want to add a documentation file with interactions with standard Sage, why not? Or is it just a flood waiting to get in?

[jdemeyer]

Replying to vdelecroix:

Do you consider your solution as temporary?

No.

I am afraid that it would become an open door to have thousands of packages in piprules.

which is problem because...? Besides, we didn't have "thousands" of optional/experimental packages before, so I don't see why this would suddenly become thousands now.

Shouldn't we reorient users toward sage -pip install X?

I don't think so, because how is the user supposed to know if he should use sage -i PKG or sage --pip install PKG? With my patch here, the user just needs to know sage -i PKG.

[jdemeyer]

Replying to tscrim:

Could this also be modified to work for nzmath and pyx?

If you successfully installed any of those two using pip, please tell me how. The obvious sage --pip install pyx or sage --pip install nzmath does not work.

[tscrim]

Vincent was able to install them, although you need to pass additional arguments to pip. See ​this thread. I will try later today to see if I can get those to work.

However, perhaps nzmath should (continue to) be a spkg as the latest version is 1.2 and pip would install 1.1?

[vdelecroix]

Seems to not work anymore:

sage -pip install --allow-external pyx --allow-unverified pyx pyx

ended up with

No files/directories in /tmp/pip-build-d6xYvF/pyx/pip-egg-info (from PKG-INFO)

and

sage -pip install --allow-external nzmath --allow-unverified nzmath nzmath

ended up with some timeout

[vdelecroix]

I just tested all the list of packages provied in comment:5 and everything worked fine.

Replying to jdemeyer:

Replying to vdelecroix:

Do you consider your solution as temporary?

No.

Then it should be complient with sage --optional which lists the packages. Right now (after installing all the packages supported) I got:

$ sage --optional
beautifulsoup........................... 3.2.1 (not_installed)
biopython............................... 1.61 (not_installed)
brian................................... 1.4.1.p0 (not_installed)
mpi4py.................................. 1.2.2 (not_installed)
trac.................................... 0.11.5.p0 (not_installed)
etc

And also with sage -t that uses optional packages.

$ sage -t whatever
...
Using --optional=cbc,mpir,python2,sage,scons
...

I am afraid that it would become an open door to have thousands of packages in piprules.

which is problem because...? Besides, we didn't have "thousands" of optional/experimental packages before, so I don't see why this would suddenly become thousands now.

Because it becomes now very easy. But you might be right.

Shouldn't we reorient users toward sage -pip install X?

I don't think so, because how is the user supposed to know if he should use sage -i PKG or sage --pip install PKG? With my patch here, the user just needs to know sage -i PKG.

No. And as Karl-Dieter said it would be good to add tests for them # optional - biopython.

Vincent

[jdemeyer]

Sorry Vincent, but that's currently too much to ask. It's hard to make these "real" packages without major changes to the package system.

So either this ticket get accepted as-is (potentially with minor modifications) or closed as "wontfix".

[jdemeyer]

Think of these packages as forming a new category: we have standard packages, optional packages, experimental packages and pip packages. They are not optional packages and cannot be listed by some sage --whatever command.

[jhpalmieri]

Although we can check existing optional packages to see if they've been installed by pip:

src/bin/sage-list-packages

@@ -5 +5 @@
 
 from __future__ import print_function
 
-import os, sys, argparse, re
+import os, argparse, re, subprocess
 from string import join
 
 if "SAGE_ROOT" not in os.environ:
@@ -98 +98 @@
 # Any other categories
 else:
     if args['version']:
+        try:
+            pip_output = subprocess.check_output(['pip', 'list'], stderr=subprocess.STDOUT)
+        except CalledProcessError:
+            pip_output = ''
+
         # Display the columns' name
         if not args['dump']:
             print("{:.<40} {} ({})\n".format("[package] ", "[latest version]",
                                              "[installed version]"))
         for p in sorted(set(local).union(remote)):
+            if p in installed:
+                installed_version = installed[p]
+            else:
+                s = re.search('\n{}.*\((.*)\)\n'.format(p), pip_output)
+                if s:
+                    installed_version = s.groups(0)[0]
+                else:
+                    installed_version = 'not_installed'
             line = [p, local.get(p, remote.get(p, 'not_found')),
-                    installed.get(p, 'not_installed')]
+                    installed_version]
             if args['dump']:
                 print(join(line, ' '))
             else:

(This is not a complete patch, just an idea.)

[jhpalmieri]

And if we think this sort of patch is a good idea, it can go on a later ticket, not this one.

[jhpalmieri]

See #19213.

[vdelecroix]

Replying to jdemeyer:

Think of these packages as forming a new category: we have standard packages, optional packages, experimental packages and pip packages. They are not optional packages and cannot be listed by some sage --whatever command.

What is this piprules file then!? It is possible to check the last version of a package with their json api

import urllib2
import json
def pip_versions(package_name):
    url = "https://pypi.python.org/pypi/{}/json".format(package_name)
    try:
        data = json.load(urllib2.urlopen(urllib2.Request(url)))
    except urllib2.HTTPError:
        return []
    else:
        versions = data["releases"].keys()
        return versions

[vdelecroix]

Replying to jdemeyer:

Sorry Vincent, but that's currently too much to ask. It's hard to make these "real" packages without major changes to the package system.

So either this ticket get accepted as-is (potentially with minor modifications) or closed as "wontfix".

I think this ticket is good, but it lacks integration. I would be also happy if all of brian, trac, ... would not appear in sage --optional.

Vincent

[vdelecroix]

In other words, we could:

• remove all of brian, trac, ... from the list of optional packages
• postponed the pip listing and test support to #19213

what do you think?

[jdemeyer]

Replying to vdelecroix:

In other words, we could:

• remove all of brian, trac, ... from the list of optional packages
• postponed the pip listing and test support to #19213

what do you think?

Fine for me. Do you give positive to this ticket then?

[vdelecroix]

Should we explicit ask on sage-devel for removing them from the server?

Vincent

[jdemeyer]

Replying to vdelecroix:

Should we explicit ask on sage-devel for removing them from the server?

See #19220.

[vbraun]

Doesn't work

$ sage -f brian
[... endless noise ...]
sage --pip uninstall -y brian
You are using pip version 6.1.1, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Cannot uninstall requirement brian, not installed
Makefile:2244: recipe for target 'brian-clean' failed

Also breaks the buildbot script that assumes that build/pkgs/* has a corresponding log file etc. IMHO one dirent for one package is IMHO a better organization, stuffing multiple packages into lines of a single file doesn't scale (merge conflicts if its too popular).

[jhpalmieri]

This might be a little ugly:

build/make/install

@@ -473 +473 @@
 
     # Add a target to remove the "installed" file for "sage -f"
     echo >&5 "$PKG_NAME-clean:"
-    echo >&5 " sage --pip uninstall -y $PKG_NAME"
+    echo >&5 " sage --pip list | grep $PKG_NAME && sage --pip uninstall -y $PKG_NAME || :"
     echo >&5
 done <"$SAGE_ROOT/build/pkgs/piprules"
 

Or the same but omitting sage --pip list | grep $PKG_NAME &&.

[jhpalmieri]

Replying to vbraun:

Also breaks the buildbot script that assumes that build/pkgs/* has a corresponding log file etc. IMHO one dirent for one package is IMHO a better organization, stuffing multiple packages into lines of a single file doesn't scale (merge conflicts if its too popular).

An alternate version could introduce a new package type ("pip"?), with one directory for each of these packages, just containing type and SPKG.txt.

[vbraun]

Another benefit of making a file-per-package is that we could use them as requirements.txt files to pin versions if necessary. Or perhaps with directories as name/requirements.txt file.

[vbraun]

Less fugly solution

sage --pip install --upgrade --force-reinstall <packagename>

[vdelecroix]

Replying to jhpalmieri:

This might be a little ugly:

build/make/install

@@ -473 +473 @@
 
     # Add a target to remove the "installed" file for "sage -f"
     echo >&5 "$PKG_NAME-clean:"
-    echo >&5 " sage --pip uninstall -y $PKG_NAME"
+    echo >&5 " sage --pip list | grep $PKG_NAME && sage --pip uninstall -y $PKG_NAME || :"
     echo >&5
 done <"$SAGE_ROOT/build/pkgs/piprules"
 

Or the same but omitting sage --pip list | grep $PKG_NAME &&.

You need to omit it since pip list does list BeautifulSoup and our package name is beautifulsoup (unless you grep without case sensitivity).

[jdemeyer]

Replying to vbraun:

Also breaks the buildbot script that assumes that build/pkgs/* has a corresponding log file

Didn't you agree in #18993 that ordinary files in build/pkgs should be ignored?

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​571410a

Add rules for installing packages with pip

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​52a3cf0

Add rules for installing packages with pip

[vdelecroix]

This is good for me. Volker?

[vbraun]

Whats the point of whitelisting only specific packages? Including trac, which has no mathematical relevance nor any utility to non-developers (and for those only barely). Having special entries in the Sage package list only makes sense if there is a need for it, e.g. because they are dependencies of non-pip (optional) packages.

IMHO if the user runs sage -f packagename and packagename is not a Sage package then either

• always try to pip install packagename, or
• finish with pip search packagename and, if it finds something, tell the user that it can be installed via sage --pip

We can merge the current to appease the cargo cultists and finally get rid of some of the broken packages, but its not entirely satisfactory.