Use pip to install Python dependencies

[embray]

New tarball: ​https://pypi.python.org/packages/26/34/0f3a5efac31f27fabce64645f8c609de9d925fe2915304d1a40f544cff0e/appnope-0.1.0.tar.gz

---

Work has already been done--for example in #19187--on using pip for sage -i where appropriate. However, a normal build of sage does not use pip for installing any Python spkgs; rather, they run python setup.py install which until recently has always been *the* way to install Python packages.

I can suggest several reasons why pip should be used instead of this method. To be clear--as many people think of pip as a tool for installing packages from the internet--this is not strictly the case. pip can be used to install an already downloaded tarball, or even straight from a source tree unpacked from a tarball or checked out from VCS. The typical way to do this is to change directories to the directory containing the setup.py script and running pip install .. This still ultimately calls setup.py install, among other setup.py commands, but with the right incantations to perform a pip-style flat install. Now, why is this preferable?

1. Future-compatibility. One of the goals that the Python packaging community has been working toward over the past several years has been to decouple Python package installation from build system. Traditionally setup.py provided both a build program and an installation program. At best it will still stick around as a build program, but its use for installation is being discouraged. There is now finally a PEP (​http://legacy.python.org/dev/peps/pep-0516/), still very much in draft state, specifically concerning this decoupling. Under this new regime pip will not likely be the *only* software for installing Python packages, but the assumption can no longer be that there is a setup.py script at all.

2. pip avoids using easy_install. easy_install is an older installation program for Python packages that was bundled with setuptools--when a Python project uses setuptools in its setup.py, the setup.py install command invokes easy_install *by default*. The behavior of easy_install is to install the package as an egg archive/directory. easy_install has several disadvantages, of which this is just one, but there are a couple reasons that this alone is a problem:
   
   
   1. Every .egg requires a sys.path entry, hence difficult to debug paths that look like this:

      >>> pprint(sys.path)
      ['',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/setuptools-18.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Cython-0.23.3-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/MarkupSafe-0.23-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Jinja2-2.7.3-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/six-1.10.0-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Pygments-2.0.2-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Sphinx-1.2.2-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/zope.interface-4.1.3-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Twisted-15.5.0-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/pytz-2013b0-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Babel-2.1.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Werkzeug-0.11.2-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/speaklater-1.3-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/python_openid-2.2.5-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/itsdangerous-0.24-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Flask-0.10.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Flask_Silk-0.2-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Flask_AutoIndex-0.5-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Flask_Babel-0.9-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Flask_OpenID-1.2.5-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/Flask_OldSessions-0.10-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/webassets-0.11.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/sagenb-0.11.6.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/backports.ssl_match_hostname-3.4.0.2-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/certifi-14.5.14-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/setuptools_scm-1.7.0-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/path.py-7.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/pickleshare-0.5-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/simplegeneric-0.8.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/decorator-4.0.6-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/networkx-1.10-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/python_dateutil-2.2-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/cycler-0.9.0-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/matplotlib-1.5.0-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/jsonschema-2.4.0-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/mistune-0.5.1-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/pip-6.1.1-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/pkgconfig-1.1.0-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/singledispatch-3.4.0.3-py2.7.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages/rpy2-2.7.5-py2.7-linux-x86_64.egg',
       '/home/iguananaut/src/sagemath/sage/local/lib/python',
       '/home/iguananaut/src/sagemath/sage/local/lib/python/site_packages',
       '/home/iguananaut/src/sagemath/sage/local/lib/python27.zip',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/plat-linux2',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-tk',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-old',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-dynload',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages']
      

      With the flat install style used by pip, and now more generally favored, the sys.path, with no eggs, would be simply:

      >>> pprint(sys.path)
      ['',
       '/home/iguananaut/src/sagemath/sage/local/lib/python',
       '/home/iguananaut/src/sagemath/sage/local/lib/python27.zip',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/plat-linux2',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-tk',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-old',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/lib-dynload',
       '/home/iguananaut/src/sagemath/sage/local/lib/python2.7/site-packages']
      

      which is certainly easier to read and reason about.
      
      
   2. The egg import mechanism requires writing to a common file during egg installation--easy-install.pth. This causes problems in parallel installations, requiring patches to setuptools. The flat install style used by pip has no such problems--there is a no single common file written to by pip during installation of packages (other than maybe the log file, which is only for debugging purposes).

As I explained in ​this comment, when a Python package is installed with pip, pip actually slips setuptools into the installation process, whether the package uses it explicitly or not. There are reasons for this not worth going into now. This is no problem for the vast majority of cases. The only exception I know of is that setuptools doesn't handle the data_files option well when --prefix is changed. As far as I can tell this is only a problem for sage itself and possibly for sagetex, but these don't even use setuptools in the first place, so until we have a workaround for that it does no harm if we keep using setup.py install for those packages, if nothing else works.

Most of the rest of the Python packages included by default in a Sage build (I haven't checked all of -optional yet) already use setuptools. I checked the setup.py of all the rest that don't explicitly use setuptools--these include docutils, mpmath, pexpect, ptyprocess, Pycrypto, pyparsing, pyzmq, scipy, and all packages from the Jupyter team (ip*, jupyter*, np*, notebook, traitlets).

Most of these packages are known by me (from personal experience) to work fine with pip. Looking at the setup.py scripts of the rest of them, most of them are fairly trivial and should work fine.

TL;DR, with the exceptions of sage and sagetex (pending work like #20108), as well as pip itself the spkg-install scripts for Python packages should change setup.py install commands to pip install .. Their dependencies would also have to be updated to require pip to be installed first.

At the same time the "uninstall" commands in those scripts can be changed to use pip uninstall.

[embray]

To be clear, I'm working on a patch for this.

[tmonteil]

IIRC, there was an issue with using pip when python was not compiled with openssl, even for building wihthout downloading. I am not sure that it is still the case, but this particular case should be taken into account since openssl (libssl-dev) is not necessarily available on the system (and cannot be made standard Sage package because of some licensing issues).

[vbraun]

Its exactly as Thierry says. Hopefully it won't be long until the announced switch of OpenSSL to a GPL-compatible license happens.

[embray]

I've attached a branch containing the beginnings of this work. I've already put it through several rounds of testing locally including installing, reinstalling packages, uninstalling and reinstalling, etc.

But I'm interested to find out what happens in the patchbot.

I haven't addressed the SSL issue yet but I will. I wanted to start getting some test / review feedback on the main work in the meantime.

---

New commits:

​9322696

Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218

[embray]

Looks like I need to rebase--that's what I get for starting a patch and then not getting around to finishing it until weeks later :)

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7447ef2

Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218

[jdemeyer]

This isn't correct:

Sage's Python unconditionally removes . from sys.path
which can break several packages' installation

Sage's Python only removes . from sys.path if having . in sys.path might be insecure. If packages break because of this, those packages should be fixed instead of working around the security check.

[jdemeyer]

In case you think that this whole security-thing is exaggerated: I can take over the Unix account of anybody who runs the Python test-suite on a computer where I have an account. When I reported this bug to Python, they didn't really care. When I reported this to IPython (which had a similar bug), they fixed their test-suite.

[embray]

As someone who has actually surreptitiously taken over Unix accounts, I'm not too impressed by that particular issue. But that's beside the point--I'm not arguing with that right now.

I'll have to more closely look at what that patch does, but I'll change that comment to not supply inaccurate information. No packages are broken here though. The reason this is needed is that pip copies files packages to a tmp dir to build them. I'm actually surprised this hasn't come up for sage -pip, though it's only an issue for packages that import themselves in their setup.py so maybe it just coincidentally hasn't come up.

[jdemeyer]

Replying to embray:

The reason this is needed is that pip copies files packages to a tmp dir to build them.

It doesn't hurt if it's in a private directory inside /tmp, say /tmp/my-pip-directory where /tmp/my-pip-directory has reasonable permissions. So I don't fully understand the problem.

[embray]

Indeed it should be doing that. It's not like it's putting the package directly under /tmp. It belongs to me and my user-specific group. So I'm not sure why there would be a problem and yet I am getting the warning about '.' being removed from sys.path and the import subsequently fails.

[jdemeyer]

Replying to embray:

Indeed it should be doing that. It's not like it's putting the package directly under /tmp. It belongs to me and my user-specific group. So I'm not sure why there would be a problem and yet I am getting the warning about '.' being removed from sys.path and the import subsequently fails.

What is the exact warning message?

[embray]

Nevermind--I traced this to a bug in pip, combined with a possible minor defect in sage-uncompress-spkg (or Python's tarfile module depending on your perspective), along with strange packaging for pyparsing, which is the package where this came up originally.

1) It's true that pip creates a temp dir (under /tmp, typically) in which to build a package. This directory is created with mkdtemp and normally has safe permissions on it. However, when installing a package from a source directory (a la pip install .), pip then deletes that temp directory, and copies the source directory using shutil.copytree to the same path as the just-deleted temp dir. There are a couple problems with this:

a) Deleting and recreating the temp dir of the same name involves a possible race condition. b) More importantly, while mkdtemp will create a directory with safe permissions, shutil.copytree copies the original permissions of the source files, which may be arbitrary. Instead, pip should be copying the contents of the source directory into the directory created with mkdtemp.

2) sage-uncompress-spkg just calls TarFile.extractall() which preserves the original permissions of all files/directories in the tar file, and doesn't provide a clear way to enable or disable this. This is different from the GNU tar command which by default (for normal users) applies the user's umask to all extracted files, which is generally safer depending on the user's umask. It would probably be best if sage-uncompress-spkg did this as well.

3) The top-level directory in pyparsing's source package has permissions set to 777. Of all the upstream source tarballs I have pyparsing is the only one like this. All the rest have 775 or 755.

Result: pyparsing's source is extracted with permissions 777 and is then copied into a directory in /tmp with unrestricted access, from which pip installs it. My previous comments aside these are all bad and all three should be addressed.

[jdemeyer]

4) python setup.py sdist doesn't actually try to create a tarball with sane permissions in the first place.

[embray]

python setup.py sdist doesn't actually try to create a tarball with sane permissions in the first place.

Agreed.... is what I was about to write. But I tested it and ./setup.py sdist does actually use sane permissions. So either the maintainer of pyparsing is not using ./setup.py sdist or is using a buggy version of Python or something.

[embray]

See #20481 for an attempt to address point 2 above, as it was the lowest-hanging fruit I think. The issue in pip will be easy to fix too but will have to get it accepted upstream, etc, etc.

[jdemeyer]

Replying to embray:

python setup.py sdist doesn't actually try to create a tarball with sane permissions in the first place.

Agreed.... is what I was about to write. But I tested it and ./setup.py sdist does actually use sane permissions.

I also tested it and ./setup.py sdist did not use sane permissions.

[embray]

Huh. How did you test it? I just ran chmod 777 on the parent directory and all subdirectories, then ran ./setup.py sdist. The permissions in the resulting tarball at least had my umask applied:

$ umask -S
u=rwx,g=rwx,o=rx

It's the 'w' I'm mostly concerned about. How are you defining "sane"?

[jdemeyer]

I made a project with the following files:

zzz.py:

print("zzz")

setup.py:

from setuptools import setup   # setuptools or distutils does not matter

setup(
    name="zzz",
    py_modules=["zzz"],
)

I changed the permissions of both these files to 0666 and then ran python setup.py sdist. The result:

drwxr-xr-x jdemeyer/jdemeyer 0 2016-04-21 17:38 zzz-0.0.0/
drwxr-xr-x jdemeyer/jdemeyer 0 2016-04-21 17:38 zzz-0.0.0/zzz.egg-info/
-rw-r--r-- jdemeyer/jdemeyer 1 2016-04-21 17:38 zzz-0.0.0/zzz.egg-info/dependency_links.txt
-rw-r--r-- jdemeyer/jdemeyer 177 2016-04-21 17:38 zzz-0.0.0/zzz.egg-info/PKG-INFO
-rw-r--r-- jdemeyer/jdemeyer 123 2016-04-21 17:38 zzz-0.0.0/zzz.egg-info/SOURCES.txt
-rw-r--r-- jdemeyer/jdemeyer   4 2016-04-21 17:38 zzz-0.0.0/zzz.egg-info/top_level.txt
-rw-r--r-- jdemeyer/jdemeyer 177 2016-04-21 17:38 zzz-0.0.0/PKG-INFO
-rw-rw-rw- jdemeyer/jdemeyer 113 2016-04-21 17:36 zzz-0.0.0/setup.py
-rw-r--r-- jdemeyer/jdemeyer  59 2016-04-21 17:38 zzz-0.0.0/setup.cfg
-rw-rw-rw- jdemeyer/jdemeyer  13 2016-04-21 17:28 zzz-0.0.0/zzz.py

[jdemeyer]

Merge conflict.

[embray]

But what about directories? I'm more concerned with what permissions it sets for directories than for files.

[jdemeyer]

Replying to embray:

But what about directories? I'm more concerned with what permissions it sets for directories than for files.

I propose to use the umask for everything which is in the tarfile: files, directories, ...

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​95dd938

Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218

[jdemeyer]

Also note that #14840 added lots of new Python packages, which might be installed using pip.

[jdemeyer]

Typo: insalled

[embray]

Ah, that explains why I had to rebase, and why pytz started showing up. Thanks, I'll update those as well.

[jdemeyer]

Something else to potentially worry about: how is the upgrade path from non-pip-installed packages to pip-installed packages? What if a package was installed before using either distutils or setuptools and it's now re-installed using pip?

[embray]

pip will uninstall them, mostly. If a package was installed without pip, it can't uninstall and scripts or data files that were not installed directly under site-packages/package_name(.egg-info)? because there's no install log like there is with pip.

That's no worse, and slightly better than the current situation where some packages were uninstalled first, but most weren't, and of those that were only did rm -rf site-packages/packagename. Some were clever enough to do rm -rf site-packages/packagename* but that could actually have unintended consequences. The former case is no good either because it leaves .egg-info directories, making it appear to tools that know how to process those like the package is "installed".

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​d2d0d74	Ignore directory permissions when extracting tarballs.
​4fe154a	Modified the custom TarFile to simply apply the user's umask to all files and directories, regardless what it may be--consistent with the default behavior of 'tar'.
​1c30cbe	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218

[embray]

Rebased to include latest from develop, and to include #20481. This allowed me to remove the env PYTHONPATH=. from all pip calls. The only package that currently was affected by this issue was pyparsing, and #20481 effectively works around it. There is a larger problem with pip that should be addressed, but it is no longer an immediate blocker for this issue.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​31b0f5f

This is no longer needed.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​f3b3cdf	Adds a new sage-apply-patches script.
​ba037a8	Fix accidental duplication of patch_subdir.
​b17fb08	Forgot to change this to loop over the full list when I made $patches into a list.
​cedf80a	Mathjax doesn't have a source directory to cd into, so uses a different default patch path from most packages.
​2ce575e	None of these packages currently have patches to apply.
​29bd199	Add license notice.
​dff74e5	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​07accc7	This is no longer needed.
​60dfae1	Adds patch to pip to allow it to be used without SSL support from Python

[embray]

I rebased this on top of my branch for #20692 (which itself is based on a recent version of develop) so that I could use it in applying a new patch to pip. Other than that the dependency is loose, and can be extracted if #20692 is rejected for some reason or requires significant more work.

I have also applied the necessary patch to pip so that it no longer requires SSL to work (at least for installing local packages).

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2fe1d35

Update some packages that were added since the start of this branch to use pip.

[embray]

The SSL issue was already known upstream before I made this ticket. However, there is a different issue with pip that I described ​here and that should still be reported upstream. I haven't gotten around to it yet, but I will.

[jdemeyer]

You need to increase the patchlevel of pip, otherwise the patch will not be applied.

[jdemeyer]

This branch does a lot of stuff. It's not a disaster, but I would have preferred to see this as 3 tickets:

1. Add the SSL patch to pip.
2. Use pip install to install Python packages.
3. Add dependencies to optional packages without dependency files.

[embray]

Replying to jdemeyer:

This branch does a lot of stuff. It's not a disaster, but I would have preferred to see this as 3 tickets:

1. Add the SSL patch to pip.
2. Use pip install to install Python packages.
3. Add dependencies to optional packages without dependency files.

Okay, that's fair. I can break this up.

Not exactly clear how to break up 2 and 3 though. If optional packages are going to be installed with pip, they have to have dependency files listing at least pip first. But if I do that before making them installed with pip then they would have an unnecessary dependency on pip.

Unless I made a separate ticket that only adds the dependency files with no pip, and then later adds the pip dependency as part of 2). But that would be a little more annoying.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e3ddedd	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.
​a30f722	Bump pip package patch level

[jdemeyer]

Over at #20739, people are suggesting to use --no-binary :all:. I don't understand why, but maybe you do...

[embray]

That shouldn't be relevant here since this has nothing to do with installing packages from PyPI (or any other PyPI-like Python package index). That's an instruction for controlling what type of package to download if there are multiple options.

[jdemeyer]

Right, makes sense.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​2fd9cf6	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​985790a	This is no longer needed.
​0a55518	Update some packages that were added since the start of this branch to use pip.
​7082445	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.

[embray]

Rebased, and simplified to no longer depend on #20692, and also removed the patch to pip for SSL, which has been broken out to #20913.

This doesn't have a strict dependence on #20913, but #20913 should still be merged first since otherwise this patch will break the sage build on platforms with no OpenSSL support.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​756d94a	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​70135d4	This is no longer needed.
​bd0e2ca	Update some packages that were added since the start of this branch to use pip.
​f028f6f	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.

[embray]

Rebased on recent develop branch.

It would be great to get this moving--I just a build fail on trying to install flask_babel (though it could have been any number of packages) because they all use setuptools, and hence running ./setup.py install makes them use easy_install by default and check online for the package's dependencies even though it shouldn't have to.

This patch ensures that Python packages are installed without dependency on PyPI.

[jdemeyer]

Merge conflict.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​0676708	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​cadba88	This is no longer needed.
​3e014d9	Update some packages that were added since the start of this branch to use pip.
​7880aca	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.

[embray]

Rebased again.

---

New commits:

​0676708	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​cadba88	This is no longer needed.
​3e014d9	Update some packages that were added since the start of this branch to use pip.
​7880aca	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.

[jdemeyer]

Volker, it would be good if you could test on the buildbot that this ticket works when building from scratch.

[jdemeyer]

Sorry for the trouble again, but this conflicts with #19295. I guess you should rebase it to that ticket. Alternatively, wait for the new Sage beta release.

Apart from that, I see no obvious issue with this ticket. It mostly needs testing. I will try to do that in the following days.

[embray]

I mean, so long as #19295 is merged I can rebase this again on develop right?

[jdemeyer]

Replying to embray:

I mean, so long as #19295 is merged I can rebase this again on develop right?

...when the next beta/rc release comes out which includes #19295, yes.

Or you just rebase right now on the branch of #19295.

[jdemeyer]

As far as I can tell, this ticket works fine. So the only obstruction is #19295.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​3a90e69	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​c028d41	This is no longer needed.
​83061e1	Update some packages that were added since the start of this branch to use pip.
​0c899a4	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.

[embray]

Rebased

[embray]

I actually need to modify this slightly. The $PIP_INSTALL variable that's set in build/make/install is now needed for running the spkg-install for all the Python packages that use it. This means spkg-install can't be run directly and *must* go through the build/make/install script.

Instead, $PIP_INSTALL should probably be set in sage-env.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​8df4b5f

Move PIP_INSTALL into sage-env so that it's available so long as the Sage environment is activated

[embray]

Done. Seems to work fine.

---

New commits:

​8df4b5f

Move PIP_INSTALL into sage-env so that it's available so long as the Sage environment is activated

[vbraun]

Fails for me when building after "make distclean" with

sagenb-export 2.0 is already the active version in easy-install.pth
Installing sagenb-export script to /mnt/disk/home/release/Sage/local/bin

Installed /mnt/disk/home/release/Sage/local/lib/python2.7/site-packages/sagenb_export-2.0-py2.7.egg
Processing dependencies for sagenb-export==2.0
Searching for entrypoints
Reading https://pypi.python.org/simple/entrypoints/

The buildbot is not allowed to download packages.

The nbconvert package lists entrypoints in local/lib/python2.7/site-packages/nbconvert-4.2.0-py2.7.egg-info/requires.txt, but only with this ticket. Without this ticket, the depenency is not picked up.

Perhaps the easiest solution would be to package entrypoints...

[embray]

sagenb_export isn't even a sage package in this branch. It must have been added since I last rebased. I mean, until the commit I just pushed it wasn't even being installed with pip so I don't know what the relation would be.

Except that whoever added sagenb_export as an spkg did not also add all its dependencies which unfortunately you have to do.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​c268094	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​a5cfec2	This is no longer needed.
​aaf7a8c	Update some packages that were added since the start of this branch to use pip.
​ec6ca6c	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.
​820314e	Move PIP_INSTALL into sage-env so that it's available so long as the Sage environment is activated
​de5cc9d	pip install sagenb_exports

[jdemeyer]

This is proving to be difficult to do all at once. Maybe this should be split up to "provide the base functionality" like the PIP_INSTALL stuff and then we can fix individual packages or batches of packages in follow-up tickets.

Then a comment about the implementation: making PIP_INSTALL an environment variable looks strange. Maybe a small script build/bin/pip-install would make more sense?

[embray]

I don't think making PIP_INSTALL an environment variable "looks strange" any more than CC does.

How is it proving to be too difficult? It works for me.

[embray]

Ah, looks like it needs to be rebased again. No problem.

[jdemeyer]

Replying to embray:

How is it proving to be too difficult?

Difficult to merge in the sense that it constantly conflicts with other tickets.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6c9dffe	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​63cade4	This is no longer needed.
​a692a30	Update some packages that were added since the start of this branch to use pip.
​3e16e00	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.
​332d915	Move PIP_INSTALL into sage-env so that it's available so long as the Sage environment is activated
​d6121b2	pip install sagenb_exports
​d7218a0	Three new Python packages were added that should use pip for installation

[embray]

Another question that comes to mind: Where would be the best place in the documentation to document how an spkg for a Python package should be installed? That should be updated ASAP to demonstrate using pip to install the package instead of the old-fashioned python setup.py install?

Anyways the sooner we get this merged, and update the documentation, the less "difficult" this will be.

---

New commits:

​6c9dffe	Use pip to install most Python-based dependencies; see http://trac.sagemath.org/ticket/20218
​63cade4	This is no longer needed.
​a692a30	Update some packages that were added since the start of this branch to use pip.
​3e16e00	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.
​332d915	Move PIP_INSTALL into sage-env so that it's available so long as the Sage environment is activated
​d6121b2	pip install sagenb_exports
​d7218a0	Three new Python packages were added that should use pip for installation

[jdemeyer]

Just a comment: I need this to fix the installation of widgetsnbextension, see #21256.

[jdemeyer]

Is this ready for review?

[jdemeyer]

Replying to jdemeyer:

Just a comment: I need this to fix the installation of widgetsnbextension, see #21256.

And it needs the --ignored-installed option that you are adding here, otherwise the broken installation remains broken.

[jdemeyer]

I do not like the --no-deps flag: it used to be an error if dependencies were not installed, now you will silently have a broken installation.

[jdemeyer]

I also like --verbose: it actually shows the output from setup.py instead of

    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: still running...
    Running setup.py install for scipy: finished with status 'done'

[embray]

Replying to jdemeyer:

I do not like the --no-deps flag: it used to be an error if dependencies were not installed, now you will silently have a broken installation.

That's a problem that Sage's package distributions should be handling. In other words, you're asking pip and PyPI to handle dependency resolution for you. But if Sage wants to maintain its own package distribution system it's its responsibility to handle dependency resolution correctly.

[embray]

I'm fine with adding the --verbose flag but -1 on removing --no-deps.

---

New commits:

​c728bd8

Remove --no-deps and add --verbose flag to pip

[jdemeyer]

Replying to embray:

That's a problem that Sage's package distributions should be handling. In other words, you're asking pip and PyPI to handle dependency resolution for you.

Not really. I want dependency checking, not dependency resolution.

It is completely normal for a package to check its dependencies upon installation: GNU configure scripts usually do this.

The flag --no-index does exactly that: it will not install dependencies but it will still check for dependencies.

[embray]

Okay, as long as it isn't recursive--that is, as long as it doesn't also check the dependencies for the dependencies--you're right that --no-index seems to ensure that.

[jdemeyer]

Hmm, removing --no-deps breaks a lot of Jupyter stuff since they have setup.py containing stuff like

setuptools_args = {}
install_requires = setuptools_args['install_requires'] = [
    'ipython>=4.0.0',
    'traitlets',
    'jupyter_client',
    'tornado>=4.0',
]

if 'setuptools' in sys.modules:
    setup_args.update(setuptools_args)

These dependencies are not seen when installing using python setup.py install.

[jdemeyer]

I will try to fix those dependencies.

[jdemeyer]

Hmm, traitlets thinks that ipython_genutils is not installed but in reality it is installed (and it is properly listed as dependency in Sage):

[traitlets-4.2.2] Processing /usr/local/src/sage-git/local/var/tmp/sage/build/traitlets-4.2.2/src
[traitlets-4.2.2]   Running setup.py (path:/tmp/pip-TnBahx-build/setup.py) egg_info for package from file:///usr/local/src/sage-git/local/var/tmp/sage/build/traitlets-4.2.2/src
[traitlets-4.2.2]     Running command python setup.py egg_info
[traitlets-4.2.2]     running egg_info
[traitlets-4.2.2]     creating pip-egg-info/traitlets.egg-info
[traitlets-4.2.2]     writing requirements to pip-egg-info/traitlets.egg-info/requires.txt
[traitlets-4.2.2]     writing pip-egg-info/traitlets.egg-info/PKG-INFO
[traitlets-4.2.2]     writing top-level names to pip-egg-info/traitlets.egg-info/top_level.txt
[traitlets-4.2.2]     writing dependency_links to pip-egg-info/traitlets.egg-info/dependency_links.txt
[traitlets-4.2.2]     writing manifest file 'pip-egg-info/traitlets.egg-info/SOURCES.txt'
[traitlets-4.2.2]     warning: manifest_maker: standard file '-c' not found
[traitlets-4.2.2] 
[traitlets-4.2.2]     reading manifest file 'pip-egg-info/traitlets.egg-info/SOURCES.txt'
[traitlets-4.2.2]     writing manifest file 'pip-egg-info/traitlets.egg-info/SOURCES.txt'
[traitlets-4.2.2]   Source in /tmp/pip-TnBahx-build has version 4.2.2, which satisfies requirement traitlets==4.2.2 from file:///usr/local/src/sage-git/local/var/tmp/sage/build/traitlets-4.2.2/src
[traitlets-4.2.2] Collecting ipython_genutils (from traitlets==4.2.2)
[traitlets-4.2.2]   0 location(s) to search for versions of ipython-genutils:
[traitlets-4.2.2]   Could not find a version that satisfies the requirement ipython_genutils (from traitlets==4.2.2) (from versions: )
[traitlets-4.2.2] Cleaning up...
[traitlets-4.2.2]   Removing source in /tmp/pip-TnBahx-build
[traitlets-4.2.2] No matching distribution found for ipython_genutils (from traitlets==4.2.2)
[traitlets-4.2.2] Exception information:
[traitlets-4.2.2] Traceback (most recent call last):
[traitlets-4.2.2]   File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/basecommand.py", line 215, in main
[traitlets-4.2.2]     status = self.run(options, args)
[traitlets-4.2.2]   File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/commands/install.py", line 299, in run
[traitlets-4.2.2]     requirement_set.prepare_files(finder)
[traitlets-4.2.2]   File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_set.py", line 370, in prepare_files
[traitlets-4.2.2]     ignore_dependencies=self.ignore_dependencies))
[traitlets-4.2.2]   File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_set.py", line 522, in _prepare_file
[traitlets-4.2.2]     finder, self.upgrade, require_hashes)
[traitlets-4.2.2]   File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_install.py", line 268, in populate_link
[traitlets-4.2.2]     self.link = finder.find_requirement(self, upgrade)
[traitlets-4.2.2]   File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/index.py", line 491, in find_requirement
[traitlets-4.2.2]     'No matching distribution found for %s' % req
[traitlets-4.2.2] DistributionNotFound: No matching distribution found for ipython_genutils (from traitlets==4.2.2)

[jdemeyer]

Hmm, could it be that the --ignore-installed option also ignores installed dependencies?

[embray]

I'll look into it.

[jdemeyer]

I tried --upgrade --force-reinstall instead of --ignore-installed but that gives the same problem:

$ pip install --verbose --no-index --upgrade --force-reinstall .
Ignoring indexes: https://pypi.python.org/simple
Processing /usr/local/src/sage-git/tmp/traitlets-4.2.2
  Running setup.py (path:/tmp/pip-Eh8Et4-build/setup.py) egg_info for package from file:///usr/local/src/sage-git/tmp/traitlets-4.2.2
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/traitlets.egg-info
    writing requirements to pip-egg-info/traitlets.egg-info/requires.txt
    writing pip-egg-info/traitlets.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/traitlets.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/traitlets.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/traitlets.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/traitlets.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/traitlets.egg-info/SOURCES.txt'
  Source in /tmp/pip-Eh8Et4-build has version 4.2.2, which satisfies requirement traitlets==4.2.2 from file:///usr/local/src/sage-git/tmp/traitlets-4.2.2
Collecting ipython_genutils (from traitlets==4.2.2)
  0 location(s) to search for versions of ipython-genutils:
  Could not find a version that satisfies the requirement ipython_genutils (from traitlets==4.2.2) (from versions: )
Cleaning up...
  Removing source in /tmp/pip-Eh8Et4-build
No matching distribution found for ipython_genutils (from traitlets==4.2.2)
Exception information:
Traceback (most recent call last):
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/commands/install.py", line 299, in run
    requirement_set.prepare_files(finder)
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_set.py", line 370, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_set.py", line 522, in _prepare_file
    finder, self.upgrade, require_hashes)
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_install.py", line 268, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pip-8.1.2-py2.7.egg/pip/index.py", line 491, in find_requirement
    'No matching distribution found for %s' % req
DistributionNotFound: No matching distribution found for ipython_genutils (from traitlets==4.2.2)

[embray]

Yes, you're right. It seems that --ignore-installed also ignores installed dependencies. The fact that I also had --no-deps hid that. That's disconcerting. It looks like you can't do dependency checking the way you wanted after all, though I'll admit that's unfortunate. You'd think it should work. I might treat that as a defect in pip.

But I don't think it's a showstopper. I still think it should be the responsibility of the sage packager to work this out and not blindly add new packages without making sure all its dependencies are correct too.

[jdemeyer]

Replying to embray:

I still think it should be the responsibility of the sage packager to work this out and not blindly add new packages without making sure all its dependencies are correct too.

I would love this to become official Sage policy. The main culprit is mass-update tickets like #20625.

[embray]

A relevant ticket in pip: ​https://github.com/pypa/pip/issues/3261

It's mentioned there that there's also a new pip check command that will do a sanity-check on the system, so running that could also be policy (without necessarily requiring the check on every package install).

[jdemeyer]

It seems that pip is missing an option to force-reinstall a single package without recursion.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​fd131c1

Add --verbose flag to pip

[jdemeyer]

OK, undid the removal of --no-deps.

I would be fine with merging this now, but still thinking about future improvements.

[embray]

Replying to jdemeyer:

It seems that pip is missing an option to force-reinstall a single package without recursion.

pip install --upgrade --force --no-deps does it. I've used that in the past.

[jdemeyer]

Replying to embray:

Replying to jdemeyer:

It seems that pip is missing an option to force-reinstall a single package without recursion.

pip install --upgrade --force --no-deps does it.

--upgrade does recursion:

  -U, --upgrade               Upgrade all specified packages to the newest available version. This process is recursive regardless of whether a dependency is already
                              satisfied.

Edit: I see you added --no-deps but I don't want that since I do want dependency checking.

[embray]

Well, yes, but now we know that dependency checking in the sense that you want doesn't work anyways.

[jdemeyer]

Replying to embray:

Well, yes, but now we know that dependency checking in the sense that you want doesn't work anyways.

It works by itself, but not in combination with the other options.

[jdemeyer]

Something like

pip uninstall PACKAGE
pip install --no-index .

would work and correctly check dependencies. There is just not a single command which does that.

[jdemeyer]

For reasons that I do not understand, this breaks running Jupyter from the command line:

$ jupyter notebook
Traceback (most recent call last):
  File "/usr/local/src/sage-git/local/bin/jupyter-notebook", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/local/src/sage-git/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'terminado>=0.3.3' distribution was not found and is required by notebook

[jdemeyer]

The obvious solution is to add a terminado package to Sage and have notebook depend on it.

[jdemeyer]

Alternative and simpler solution: keep using distutils instead of pip to install notebook.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​9b13184

Install the Jupyter notebook using distutils to avoid dependency on terminado

[jdemeyer]

This is exactly the kind of problem which can happen because of --no-deps: notebook has an unsatisfied dependency (terminado). This breaks stuff but might pass automated tested since actually running jupyter is not tested.

[embray]

I'm not sure I'm following you. Obviously jupyter notebook has a dependency that isn't satisfied because Sage doesn't include it. Just forcing it to install without that dependency and leaving it broken is not a solution.

[jdemeyer]

Replying to embray:

Obviously jupyter notebook has a dependency that isn't satisfied because Sage doesn't include it. Just forcing it to install without that dependency and leaving it broken is not a solution.

I would argue that the bug is in Jupyter's requirements list: it lists a dependency which in practice is not a dependency at all, but more like an optional add-on.

I just want to get on with this and this is a simple solution which is known to work.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​feed1dc

Add a few missing dependencies

[jdemeyer]

This last commit adds a few missing dependencies, exposed by #21256.

[embray]

To be fair, terminado supplies the terminal interface in the Notebook, so even if as you're claiming it's optional, it is certainly nice to have. In any case you can say it's a bug or not but if they're including it in their requirements list then I say it's a requirement unless specified otherwise. This is definitely no solution.

[jdemeyer]

So what we should then? I see two options:

1. Patch notebook/setup.py to remove terminado from the dependencies.

2. Add a terminado package to Sage as dependency of this ticket.

[embray]

I suggest option 2. As it is, for the Docker image for Sage I'm also installing terminado separately. I don't see why it shouldn't be included.

If it helps, looking at the setup.py for notebook, it seems terminado is being considered a requirement on any non-Windows platform. Probably because even though they've been careful to make it so that the notebook can work without it, not having it means missing functionality==a defect. I don't think we should say that "Sage includes the Jupyter notebook" and then ship a defective product.

The reason you get this

pkg_resources.DistributionNotFound: The 'terminado>=0.3.3' distribution was not found and is required by notebook

is that installing with setuptools also installs the autogenerated console script wrappers which do dependency checking when you run them, so it's right to say that it won't run without some dependency installed.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​6c47f0b

Add a few missing dependencies

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​afc90d4	Add terminado as standard package
​115e77b	Merge remote-tracking branch 'trac/u/jdemeyer/add_a_terminado_package' into t/20218/pip-install
​829a5b4	Add a few missing dependencies

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​fdb403e	Add entrypoints and configparser packages
​c4fc97c	Merge remote-tracking branches 'trac/u/jdemeyer/add_a_terminado_package' and 'trac/u/jdemeyer/add_a_entrypoints_package' into t/20218/pip-install
​1e0a377	Fix installation of Jupyter packages

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​7054c41

Fix installation of Jupyter packages

[embray]

This is minor, but a few packages (configparser, terminado, entrypoints) still have an explicit pip install in their spgk-install instead of the $PIP_INSTALL. The effect is the same, but if it's not fixed now but the idea was to have a standard way of invoking pip for all packages, and it could have surprising effects if some packages aren't getting the same arguments for pip.

[jdemeyer]

That can be fixed in a follow-up ticket. There is no reason to hold up this ticket for that.

[vbraun]

Breaks OSX when compiling from scratch in numpy

[numpy-1.11.1] gcc version 4.9.3 (GCC) 
[numpy-1.11.1] ****************************************************
[numpy-1.11.1] patching file numpy/core/function_base.py
[numpy-1.11.1] patching file numpy/distutils/system_info.py
[numpy-1.11.1] Hunk #1 succeeded at 1685 (offset -5 lines).
[numpy-1.11.1] Hunk #2 succeeded at 1718 (offset -4 lines).
[numpy-1.11.1] 
[numpy-1.11.1] Usage:   
[numpy-1.11.1]   pip install [options] <requirement specifier> [package-index-options] ...
[numpy-1.11.1]   pip install [options] -r <requirements file> [package-index-options] ...
[numpy-1.11.1]   pip install [options] [-e] <vcs project url> ...
[numpy-1.11.1]   pip install [options] [-e] <local project path> ...
[numpy-1.11.1]   pip install [options] <archive url/path> ...
[numpy-1.11.1] 
[numpy-1.11.1] no such option: ---global-option
[numpy-1.11.1] 
[numpy-1.11.1] real	0m0.519s
[numpy-1.11.1] user	0m0.269s
[numpy-1.11.1] sys	0m0.142s
[numpy-1.11.1] ************************************************************************
[numpy-1.11.1] Error installing package numpy-1.11.1

Note on OSX:

osx:build buildslave-sage$ NUMPY_FCONFIG="config_fc --noopt --noarch"
osx:build buildslave-sage$ echo $NUMPY_FCONFIG | sed -e 's/\(\S*\)/--global-option \1/g'
--global-option c--global-option o--global-option n--global-option f--global-option i--global-option g--global-option _--global-option f--global-option c--global-option  --global-option ---global-option ---global-option n--global-option o--global-option o--global-option p--global-option t--global-option  --global-option ---global-option ---global-option n--global-option o--global-option a--global-option r--global-option c--global-option h--global-option 

[embray]

Apparently on OSX that should be sed -E not sed -e :(

[embray]

Replying to embray:

Apparently on OSX that should be sed -E not sed -e :(

In fact the -e doesn't really even need to be there. I'm not sure why I put it. But on OSX it seems the -E is needed to interpret the command as a "modern" perl-style regexp.

[embray]

Or for a more portable way without invoking sed at all:

$(for opt in $NUMPY_FCONFIG; do printf ' %s %s' --global-option $opt; done)

[vbraun]

I don't really care which variant long as it works ;-)

---

Last 10 new commits:

​a692a30	Update some packages that were added since the start of this branch to use pip.
​3e16e00	Some of the packages changed to use pip in 2fe1d357600e60d3422141ba5a7fb16efb3a691b needed pip added to their build deps.
​332d915	Move PIP_INSTALL into sage-env so that it's available so long as the Sage environment is activated
​d6121b2	pip install sagenb_exports
​d7218a0	Three new Python packages were added that should use pip for installation
​fd131c1	Add --verbose flag to pip
​afc90d4	Add terminado as standard package
​fdb403e	Add entrypoints and configparser packages
​c4fc97c	Merge remote-tracking branches 'trac/u/jdemeyer/add_a_terminado_package' and 'trac/u/jdemeyer/add_a_entrypoints_package' into t/20218/pip-install
​7054c41	Fix installation of Jupyter packages

[jdemeyer]

Is this ready for review?

[jdemeyer]

I guess not, since no changes have been made :-)

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​99e1494

Revert changes to numpy

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f5c0b16

wcwidth depends on pip

[embray]

Replying to git:

Branch pushed to git repo; I updated commit sha1. New commits:

​99e1494

Revert changes to numpy

I mean that's fine--what this is doing is in effect the same. But you could have taken one of my suggestions above. I think the for loop is most effective. I mean I would just do it but you sort of took over this ticket

[jdemeyer]

I just want this ticket to get merged. We can deal with the difficult cases (like numpy) later.

[jdemeyer]

Replying to embray:

I mean that's fine

May I take that as positive review?

[embray]

Yeah, yeah :) *fingers crossed*

[vbraun]

IPython on OSX appears to require a platform-specific package:

sage -t --long src/sage/tests/cmdline.py
**********************************************************************
File "src/sage/tests/cmdline.py", line 476, in sage.tests.cmdline.test_executable
Failed example:
    out.find("5559060566555523") >= 0
Expected:
    True
Got:
    False
**********************************************************************
File "src/sage/tests/cmdline.py", line 478, in sage.tests.cmdline.test_executable
Failed example:
    err
Expected:
    ''
Got:
    'Traceback (most recent call last):\n  File "/Users/buildslave-sage/slave/sage_git/build/local/bin/ipython", line 5, in <module>\n    from pkg_resources import load_entry_point\n  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>\n    @_call_aside\n  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside\n    f(*args, **kwargs)\n  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set\n    working_set = WorkingSet._build_master()\n  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master\n    ws.require(__requires__)\n  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require\n    needed = self.resolve(parse_requirements(requirements))\n  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve\n    raise DistributionNotFound(req, requirers)\npkg_resources.DistributionNotFound: The \'appnope\' distribution was not found and is required by ipython\n'
**********************************************************************
File "src/sage/tests/cmdline.py", line 480, in sage.tests.cmdline.test_executable
Failed example:
    ret
Expected:
    0
Got:
    1
**********************************************************************
1 item had failures:
   3 of 239 in sage.tests.cmdline.test_executable
    [238 tests, 3 failures, 71.27 s]

[vbraun]

​https://pypi.python.org/pypi/appnope/

[vbraun]

osx:build buildslave-sage$ ./sage --ipython
Traceback (most recent call last):
  File "/Users/buildslave-sage/slave/sage_git/build/local/bin/ipython", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2927, in <module>
    @_call_aside
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2913, in _call_aside
    f(*args, **kwargs)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2940, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 635, in _build_master
    ws.require(__requires__)
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 943, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/Users/buildslave-sage/slave/sage_git/build/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 829, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'appnope' distribution was not found and is required by ipython
osx:build buildslave-sage$ ./sage -pip install appnope
Collecting appnope
  Downloading appnope-0.1.0-py2.py3-none-any.whl
Installing collected packages: appnope
Successfully installed appnope-0.1.0
osx:build buildslave-sage$ ./sage --ipython
Python 2.7.10 (default, Aug 27 2016, 01:49:52) 
Type "copyright", "credits" or "license" for more information.

IPython 5.0.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​55f9e9d

Add appnope package

[vbraun]

sage -t --long src/sage/misc/package.py
**********************************************************************
File "src/sage/misc/package.py", line 24, in sage.misc.package
Failed example:
    sorted(pkgs.keys())
Expected:
    ['4ti2',
     'alabaster',
     'arb',
     ...
     'zlib',
     'zn_poly',
     'zope_interface']
Got:
    ['4ti2',
     'alabaster',
     'appnope',
     'arb',

[embray]

Heh. What an insidious test :)

[jdemeyer]

That test doesn't exist in this branch... all tests pass on my machine. It must have been added recently.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7f365e3

Merge tag '7.4.beta2' into t/20218/pip-install

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​f6859c3

Mark package tests as random

[jdemeyer]

That package is only used for documentation purposes, it makes no sense to hardcode the list of packages in a test. Therefore, I mark the test # random.

[jdemeyer]

Please review this trivial fix...

[jdemeyer]

Replying to embray:

pip will uninstall them, mostly.

This doesn't happen, leading to breakage like #21441.

[embray]

It does happen, normally. I don't know why it didn't for you. You might have already had 2 Cythons installed and maybe it only uninstalled one.