Update pip to 9.0.1

[mderickx]

At #20913 pip was patched so that it works without ssl support, and this patch was also submitted upstream. Upstream has since merged the patch at ​​https://github.com/pypa/pip/issues/1165 into 9.0.1. So it makes sense to upgrade pip to this new version in order to have an unpatched pip in sage.

The pip tarbal can be found at ​https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9

[mderickx]

New commits:

​84ecb82

Upgrade pip to 9.0.1 trac #23615

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​a8d1697

Removed use of decrepated pip list format: trac 23615

[jdemeyer]

This looks very scary: eval(stdout). If you want to parse JSON, I would recommend the ​json module.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b6b9954

use json for parsing json instead of eval: trac 23615

[mderickx]

Yeah, I shouldn't have written that. Changed it into json.loads(stdout).

[jhpalmieri]

Builds fine and passes tests for me on OS X. Note that on this machine, the python2 log says that ssl is not built:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               dl              
gdbm               imageop            linuxaudiodev   
ossaudiodev        spwd               sunaudiodev     

and import ssl raises an ImportError. So it's good that this new pip works.

[jdemeyer]

Replying to jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

[jhpalmieri]

Replying to jdemeyer:

Replying to jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

Build from scratch.

[chapoton]

It seems that this breaks something with python3... :(

chapoton@icj-laptop:~/sage3$ ./sage -br
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/home/chapoton/sage3/build/pkgs                \
    SAGE_CYTHONIZED=/home/chapoton/sage3/src/build/cythonized      \
&& sage-python23 -u setup.py --no-user-cfg build install

Usage:   
  pip list [options]

no such option: --format
************************************************************************
Traceback (most recent call last):
  File "setup.py", line 69, in <module>
    from module_list import ext_modules, library_order, aliases
  File "/home/chapoton/sage3/src/module_list.py", line 166, in <module>
    from sage_setup.optional_extension import OptionalExtension
  File "/home/chapoton/sage3/src/sage_setup/optional_extension.py", line 24, in <module>
    all_packages = list_packages(local=True)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 226, in list_packages
    installed = installed_packages(exclude_pip)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 286, in installed_packages
    installed.update(pip_installed_packages())
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 148, in pip_installed_packages
    return {package['name'].lower():package['version'] for package in json.loads(stdout)}
  File "/home/chapoton/sage3/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
************************************************************************
Error building the Sage library
************************************************************************
Makefile:34 : la recette pour la cible « sage » a échouée
make: *** [sage] Erreur 1

[chapoton]

oh, wait a moment. I did not rebuild..

[chapoton]

Apparently, this really did break the python3 build. I am not amused.

[mderickx]

Hi Chapoton,

I would find it weird if this really did break the python3 build. The error message you posted is one that one would get if the installed pip is not actually 9.0.1:

For example on my machine with sage before this was merged

Maartens-MacBook-Pro:sagedev mderickx$ sage -pip -V
pip 8.1.2 from /Applications/sage/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ sage -pip list --format json

Usage:   
  pip list [options]

no such option: --format

and after it is merged

Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip -V
pip 9.0.1 from /Applications/sagedev/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip list --format json
[{"version": "0.7.8", "name": "alabaster"}, ..., {"version": "4.2.0", "name": "zope.interface"}]

Could you give me the output of sage -pip -V and ls $SAGE_ROOT/upstream | grep pip?

[mderickx]

A quick solution would be:

sage -pip uninstall pip
sage -i pip

This avoids doing:

make distclean
make install

[chapoton]

chapoton@icj-laptop:~/sage3$ ./sage -pip -V
pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)
chapoton@icj-laptop:~/sage3$ ls upstream/pip*
upstream/pip-8.1.2.tar.gz  upstream/pip-9.0.1.tar.gz
chapoton@icj-laptop:~/sage3$ ./sage -pip list --format json
[{"version": "1.0.0", "name": "cypari2"}, {"version": "1.6.5", "name": "cysignals"}, {"version": "0.26", "name": "Cython"}, {"version": "9.0.1", "name": "pip"}, {"version": "33.1.1", "name": "setuptools"}]

so maybe this is a matter of pip versus pip3 ?

[mderickx]

Weird, then the only possibility is that sage is picking up some pip from outside its installation.

Could you add the lines

proc = subprocess.Popen(["pip", "-V"], stdout=subprocess.PIPE)
stdout = str(proc.communicate()[0])
print(stdout)

before the lines:

proc = subprocess.Popen(["pip", "list", "--no-index", "--format", "json"], stdout=subprocess.PIPE)
     stdout = str(proc.communicate()[0])

in src/sage/misc/package.py in order to see which pip is being picked up at the point where the build fails?

Alternatively could you describe how to reproduce this error so I can look at it myself?

[chapoton]

The result of the added print is

b'pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)\n'

To reproduce, I think this is enough:

- git clone a new sage
- export SAGE_PYTHON3=yes
- make build

[mderickx]

Ah I see the problem it is a unicode thing. If you do:

stdout = proc.communicate()[0].decode()

then everything should work. I will create a new ticket with a fix.

[mderickx]

See #23822

[chapoton]

I am hurt by all this again when going from sage 8.4.beta0 to 8.4.beta1.

[jdemeyer]

Hurt by what again?

[chapoton]

I got build failures for sagelib on the very same line

return {package['name'].lower():package['version'] for package in json.loads(stdout)}

but I have now realized that this is probably due to the existence of a personal "pip.conf".