Opened 5 years ago

Closed 5 years ago

Last modified 4 years ago

#23615 closed enhancement (fixed)

Update pip to 9.0.1

Reported by: Maarten Derickx Owned by:
Priority: major Milestone: sage-8.1
Component: packages: standard Keywords:
Cc: Merged in:
Authors: Maarten Derickx Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: b6b9954 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by Maarten Derickx)

At #20913 pip was patched so that it works without ssl support, and this patch was also submitted upstream. Upstream has since merged the patch at ​https://github.com/pypa/pip/issues/1165 into 9.0.1. So it makes sense to upgrade pip to this new version in order to have an unpatched pip in sage.

The pip tarbal can be found at https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9

Change History (25)

comment:1 Changed 5 years ago by Maarten Derickx

Description: modified (diff)

comment:2 Changed 5 years ago by Maarten Derickx

Branch: u/mderickx/23615
Commit: 84ecb824e35535288dc980fa94cff9a0fec31190

New commits:

84ecb82Upgrade pip to 9.0.1 trac #23615

comment:3 Changed 5 years ago by git

Commit: 84ecb824e35535288dc980fa94cff9a0fec31190a8d1697b4b56dbe20f8332bc1590bc86ba48d815

Branch pushed to git repo; I updated commit sha1. New commits:

a8d1697Removed use of decrepated pip list format: trac 23615

comment:4 Changed 5 years ago by Maarten Derickx

Status: newneeds_review

comment:5 Changed 5 years ago by Jeroen Demeyer

Status: needs_reviewneeds_work

This looks very scary: eval(stdout). If you want to parse JSON, I would recommend the json module.

comment:6 Changed 5 years ago by git

Commit: a8d1697b4b56dbe20f8332bc1590bc86ba48d815b6b9954bb690cad965f4ffa135cd8c46b5d8cd3f

Branch pushed to git repo; I updated commit sha1. New commits:

b6b9954use json for parsing json instead of eval: trac 23615

comment:7 Changed 5 years ago by Maarten Derickx

Status: needs_workneeds_review

Yeah, I shouldn't have written that. Changed it into json.loads(stdout).

comment:8 Changed 5 years ago by John Palmieri

Builds fine and passes tests for me on OS X. Note that on this machine, the python2 log says that ssl is not built:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               dl              
gdbm               imageop            linuxaudiodev   
ossaudiodev        spwd               sunaudiodev     

and import ssl raises an ImportError. So it's good that this new pip works.

comment:9 in reply to:  8 ; Changed 5 years ago by Jeroen Demeyer

Replying to jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

comment:10 in reply to:  9 Changed 5 years ago by John Palmieri

Replying to jdemeyer:

Replying to jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

Build from scratch.

comment:11 Changed 5 years ago by Jeroen Demeyer

Reviewers: Jeroen Demeyer
Status: needs_reviewpositive_review

comment:12 Changed 5 years ago by Volker Braun

Branch: u/mderickx/23615b6b9954bb690cad965f4ffa135cd8c46b5d8cd3f
Resolution: fixed
Status: positive_reviewclosed

comment:13 Changed 5 years ago by Frédéric Chapoton

Commit: b6b9954bb690cad965f4ffa135cd8c46b5d8cd3f

It seems that this breaks something with python3... :(

chapoton@icj-laptop:~/sage3$ ./sage -br
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/home/chapoton/sage3/build/pkgs                \
    SAGE_CYTHONIZED=/home/chapoton/sage3/src/build/cythonized      \
&& sage-python23 -u setup.py --no-user-cfg build install

Usage:   
  pip list [options]

no such option: --format
************************************************************************
Traceback (most recent call last):
  File "setup.py", line 69, in <module>
    from module_list import ext_modules, library_order, aliases
  File "/home/chapoton/sage3/src/module_list.py", line 166, in <module>
    from sage_setup.optional_extension import OptionalExtension
  File "/home/chapoton/sage3/src/sage_setup/optional_extension.py", line 24, in <module>
    all_packages = list_packages(local=True)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 226, in list_packages
    installed = installed_packages(exclude_pip)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 286, in installed_packages
    installed.update(pip_installed_packages())
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 148, in pip_installed_packages
    return {package['name'].lower():package['version'] for package in json.loads(stdout)}
  File "/home/chapoton/sage3/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
************************************************************************
Error building the Sage library
************************************************************************
Makefile:34 : la recette pour la cible « sage » a échouée
make: *** [sage] Erreur 1

comment:14 Changed 5 years ago by Frédéric Chapoton

oh, wait a moment. I did not rebuild..

comment:15 Changed 5 years ago by Frédéric Chapoton

Apparently, this really did break the python3 build. I am not amused.

comment:16 Changed 5 years ago by Maarten Derickx

Hi Chapoton,

I would find it weird if this really did break the python3 build. The error message you posted is one that one would get if the installed pip is not actually 9.0.1:

For example on my machine with sage before this was merged

Maartens-MacBook-Pro:sagedev mderickx$ sage -pip -V
pip 8.1.2 from /Applications/sage/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ sage -pip list --format json

Usage:   
  pip list [options]

no such option: --format

and after it is merged

Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip -V
pip 9.0.1 from /Applications/sagedev/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip list --format json
[{"version": "0.7.8", "name": "alabaster"}, ..., {"version": "4.2.0", "name": "zope.interface"}]

Could you give me the output of sage -pip -V and ls $SAGE_ROOT/upstream | grep pip?

comment:17 Changed 5 years ago by Maarten Derickx

A quick solution would be:

sage -pip uninstall pip
sage -i pip

This avoids doing:

make distclean
make install

comment:18 Changed 5 years ago by Frédéric Chapoton

chapoton@icj-laptop:~/sage3$ ./sage -pip -V
pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)
chapoton@icj-laptop:~/sage3$ ls upstream/pip*
upstream/pip-8.1.2.tar.gz  upstream/pip-9.0.1.tar.gz
chapoton@icj-laptop:~/sage3$ ./sage -pip list --format json
[{"version": "1.0.0", "name": "cypari2"}, {"version": "1.6.5", "name": "cysignals"}, {"version": "0.26", "name": "Cython"}, {"version": "9.0.1", "name": "pip"}, {"version": "33.1.1", "name": "setuptools"}]

so maybe this is a matter of pip versus pip3 ?

EDIT: in a sage shell, pip3 says version 9.0.1 too.

Last edited 5 years ago by Frédéric Chapoton (previous) (diff)

comment:19 Changed 5 years ago by Maarten Derickx

Weird, then the only possibility is that sage is picking up some pip from outside its installation.

Could you add the lines

proc = subprocess.Popen(["pip", "-V"], stdout=subprocess.PIPE)
stdout = str(proc.communicate()[0])
print(stdout)

before the lines:

proc = subprocess.Popen(["pip", "list", "--no-index", "--format", "json"], stdout=subprocess.PIPE)
     stdout = str(proc.communicate()[0])

in src/sage/misc/package.py in order to see which pip is being picked up at the point where the build fails?

Alternatively could you describe how to reproduce this error so I can look at it myself?

comment:20 Changed 5 years ago by Frédéric Chapoton

The result of the added print is

b'pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)\n'

To reproduce, I think this is enough:

- git clone a new sage
- export SAGE_PYTHON3=yes
- make build

comment:21 Changed 5 years ago by Maarten Derickx

Ah I see the problem it is a unicode thing. If you do:

stdout = proc.communicate()[0].decode()

then everything should work. I will create a new ticket with a fix.

comment:22 Changed 5 years ago by Maarten Derickx

See #23822

comment:23 Changed 4 years ago by Frédéric Chapoton

I am hurt by all this again when going from sage 8.4.beta0 to 8.4.beta1.

comment:24 Changed 4 years ago by Jeroen Demeyer

Hurt by what again?

comment:25 Changed 4 years ago by Frédéric Chapoton

I got build failures for sagelib on the very same line

return {package['name'].lower():package['version'] for package in json.loads(stdout)}

but I have now realized that this is probably due to the existence of a personal "pip.conf".

Note: See TracTickets for help on using tickets.