Opened 2 years ago

Closed 2 years ago

Last modified 13 months ago

#23615 closed enhancement (fixed)

Update pip to 9.0.1

Reported by: mderickx Owned by:
Priority: major Milestone: sage-8.1
Component: packages: standard Keywords:
Cc: Merged in:
Authors: Maarten Derickx Reviewers: Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: b6b9954 (Commits) Commit:
Dependencies: Stopgaps:

Description (last modified by mderickx)

At #20913 pip was patched so that it works without ssl support, and this patch was also submitted upstream. Upstream has since merged the patch at ​https://github.com/pypa/pip/issues/1165 into 9.0.1. So it makes sense to upgrade pip to this new version in order to have an unpatched pip in sage.

The pip tarbal can be found at https://pypi.python.org/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#md5=35f01da33009719497f01a4ba69d63c9

Change History (25)

comment:1 Changed 2 years ago by mderickx

  • Description modified (diff)

comment:2 Changed 2 years ago by mderickx

  • Branch set to u/mderickx/23615
  • Commit set to 84ecb824e35535288dc980fa94cff9a0fec31190

New commits:

84ecb82Upgrade pip to 9.0.1 trac #23615

comment:3 Changed 2 years ago by git

  • Commit changed from 84ecb824e35535288dc980fa94cff9a0fec31190 to a8d1697b4b56dbe20f8332bc1590bc86ba48d815

Branch pushed to git repo; I updated commit sha1. New commits:

a8d1697Removed use of decrepated pip list format: trac 23615

comment:4 Changed 2 years ago by mderickx

  • Status changed from new to needs_review

comment:5 Changed 2 years ago by jdemeyer

  • Status changed from needs_review to needs_work

This looks very scary: eval(stdout). If you want to parse JSON, I would recommend the json module.

comment:6 Changed 2 years ago by git

  • Commit changed from a8d1697b4b56dbe20f8332bc1590bc86ba48d815 to b6b9954bb690cad965f4ffa135cd8c46b5d8cd3f

Branch pushed to git repo; I updated commit sha1. New commits:

b6b9954use json for parsing json instead of eval: trac 23615

comment:7 Changed 2 years ago by mderickx

  • Status changed from needs_work to needs_review

Yeah, I shouldn't have written that. Changed it into json.loads(stdout).

comment:8 follow-up: Changed 2 years ago by jhpalmieri

Builds fine and passes tests for me on OS X. Note that on this machine, the python2 log says that ssl is not built:

Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _ssl               dl              
gdbm               imageop            linuxaudiodev   
ossaudiodev        spwd               sunaudiodev     

and import ssl raises an ImportError. So it's good that this new pip works.

comment:9 in reply to: ↑ 8 ; follow-up: Changed 2 years ago by jdemeyer

Replying to jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

comment:10 in reply to: ↑ 9 Changed 2 years ago by jhpalmieri

Replying to jdemeyer:

Replying to jhpalmieri:

Builds fine and passes tests for me on OS X.

Build from scratch or just a regular upgrade? I'm going to test a build from scratch on Linux.

Build from scratch.

comment:11 Changed 2 years ago by jdemeyer

  • Reviewers set to Jeroen Demeyer
  • Status changed from needs_review to positive_review

comment:12 Changed 2 years ago by vbraun

  • Branch changed from u/mderickx/23615 to b6b9954bb690cad965f4ffa135cd8c46b5d8cd3f
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:13 Changed 2 years ago by chapoton

  • Commit b6b9954bb690cad965f4ffa135cd8c46b5d8cd3f deleted

It seems that this breaks something with python3... :(

chapoton@icj-laptop:~/sage3$ ./sage -br
cd . && export                                    \
    SAGE_ROOT=/doesnotexist                               \
    SAGE_SRC=/doesnotexist                                \
    SAGE_SRC_ROOT=/doesnotexist                           \
    SAGE_DOC_SRC=/doesnotexist                            \
    SAGE_BUILD_DIR=/doesnotexist                          \
    SAGE_PKGS=/home/chapoton/sage3/build/pkgs                \
    SAGE_CYTHONIZED=/home/chapoton/sage3/src/build/cythonized      \
&& sage-python23 -u setup.py --no-user-cfg build install

Usage:   
  pip list [options]

no such option: --format
************************************************************************
Traceback (most recent call last):
  File "setup.py", line 69, in <module>
    from module_list import ext_modules, library_order, aliases
  File "/home/chapoton/sage3/src/module_list.py", line 166, in <module>
    from sage_setup.optional_extension import OptionalExtension
  File "/home/chapoton/sage3/src/sage_setup/optional_extension.py", line 24, in <module>
    all_packages = list_packages(local=True)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 226, in list_packages
    installed = installed_packages(exclude_pip)
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 286, in installed_packages
    installed.update(pip_installed_packages())
  File "/home/chapoton/sage3/src/sage/misc/package.py", line 148, in pip_installed_packages
    return {package['name'].lower():package['version'] for package in json.loads(stdout)}
  File "/home/chapoton/sage3/local/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/chapoton/sage3/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
************************************************************************
Error building the Sage library
************************************************************************
Makefile:34 : la recette pour la cible « sage » a échouée
make: *** [sage] Erreur 1

comment:14 Changed 2 years ago by chapoton

oh, wait a moment. I did not rebuild..

comment:15 Changed 2 years ago by chapoton

Apparently, this really did break the python3 build. I am not amused.

comment:16 Changed 2 years ago by mderickx

Hi Chapoton,

I would find it weird if this really did break the python3 build. The error message you posted is one that one would get if the installed pip is not actually 9.0.1:

For example on my machine with sage before this was merged

Maartens-MacBook-Pro:sagedev mderickx$ sage -pip -V
pip 8.1.2 from /Applications/sage/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ sage -pip list --format json

Usage:   
  pip list [options]

no such option: --format

and after it is merged

Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip -V
pip 9.0.1 from /Applications/sagedev/local/lib/python2.7/site-packages (python 2.7)
Maartens-MacBook-Pro:sagedev mderickx$ ./sage -pip list --format json
[{"version": "0.7.8", "name": "alabaster"}, ..., {"version": "4.2.0", "name": "zope.interface"}]

Could you give me the output of sage -pip -V and ls $SAGE_ROOT/upstream | grep pip?

comment:17 Changed 2 years ago by mderickx

A quick solution would be:

sage -pip uninstall pip
sage -i pip

This avoids doing:

make distclean
make install

comment:18 Changed 2 years ago by chapoton

chapoton@icj-laptop:~/sage3$ ./sage -pip -V
pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)
chapoton@icj-laptop:~/sage3$ ls upstream/pip*
upstream/pip-8.1.2.tar.gz  upstream/pip-9.0.1.tar.gz
chapoton@icj-laptop:~/sage3$ ./sage -pip list --format json
[{"version": "1.0.0", "name": "cypari2"}, {"version": "1.6.5", "name": "cysignals"}, {"version": "0.26", "name": "Cython"}, {"version": "9.0.1", "name": "pip"}, {"version": "33.1.1", "name": "setuptools"}]

so maybe this is a matter of pip versus pip3 ?

EDIT: in a sage shell, pip3 says version 9.0.1 too.

Last edited 2 years ago by chapoton (previous) (diff)

comment:19 Changed 2 years ago by mderickx

Weird, then the only possibility is that sage is picking up some pip from outside its installation.

Could you add the lines

proc = subprocess.Popen(["pip", "-V"], stdout=subprocess.PIPE)
stdout = str(proc.communicate()[0])
print(stdout)

before the lines:

proc = subprocess.Popen(["pip", "list", "--no-index", "--format", "json"], stdout=subprocess.PIPE)
     stdout = str(proc.communicate()[0])

in src/sage/misc/package.py in order to see which pip is being picked up at the point where the build fails?

Alternatively could you describe how to reproduce this error so I can look at it myself?

comment:20 Changed 2 years ago by chapoton

The result of the added print is

b'pip 9.0.1 from /home/chapoton/sage3/local/lib/python2.7/site-packages (python 2.7)\n'

To reproduce, I think this is enough:

- git clone a new sage
- export SAGE_PYTHON3=yes
- make build

comment:21 Changed 2 years ago by mderickx

Ah I see the problem it is a unicode thing. If you do:

stdout = proc.communicate()[0].decode()

then everything should work. I will create a new ticket with a fix.

comment:22 Changed 2 years ago by mderickx

See #23822

comment:23 Changed 13 months ago by chapoton

I am hurt by all this again when going from sage 8.4.beta0 to 8.4.beta1.

comment:24 Changed 13 months ago by jdemeyer

Hurt by what again?

comment:25 Changed 13 months ago by chapoton

I got build failures for sagelib on the very same line

return {package['name'].lower():package['version'] for package in json.loads(stdout)}

but I have now realized that this is probably due to the existence of a personal "pip.conf".

Note: See TracTickets for help on using tickets.