Opened 5 years ago

Closed 5 years ago

#17368 closed enhancement (fixed)

Make pip a standard package

Reported by: aapitzsch Owned by:
Priority: major Milestone: sage-6.5
Component: packages: standard Keywords:
Cc: Merged in:
Authors: Thierry Monteil Reviewers: Andrey Novoseltsev, Jeroen Demeyer
Report Upstream: N/A Work issues:
Branch: 7380e00 (Commits) Commit: 7380e00fcfecfc5d563b82ba5383cdea9b8f8ba7
Dependencies: Stopgaps:

Description (last modified by tmonteil)

Though this package is less than 1 year old, there seems to be a large consensus about this, see the following thread:

https://groups.google.com/forum/#!msg/sage-devel/e48IruDu7Kg/bKf_RL4SNk8J http://thread.gmane.org/gmane.comp.mathematics.sage.devel/71580

Change History (21)

comment:1 Changed 5 years ago by tmonteil

  • Branch set to u/tmonteil/make_pip_a_standard_package

comment:2 Changed 5 years ago by git

  • Commit set to 107e511f3857ccef93a01d2c22a6caa51b9ecf23

Branch pushed to git repo; I updated commit sha1. New commits:

107e511#17368 forgot to modify install script.

comment:3 Changed 5 years ago by tmonteil

  • Authors set to Thierry Monteil
  • Description modified (diff)
  • Status changed from new to needs_review

comment:4 Changed 5 years ago by git

  • Commit changed from 107e511f3857ccef93a01d2c22a6caa51b9ecf23 to 013e2e27978d4b48dd38ee3a00825f385286b0ba

Branch pushed to git repo; I updated commit sha1. New commits:

013e2e2#17368 remove a test related to pip being optional.

comment:5 Changed 5 years ago by novoselt

  • Reviewers set to Andrey Novoseltsev
  • Status changed from needs_review to positive_review

comment:6 follow-up: Changed 5 years ago by jdemeyer

  • Status changed from positive_review to needs_work
pip-1.5.6
====================================================
Setting up build directory for pip-1.5.6
Finished set up
****************************************************
Host system:
Linux tamiyo 3.7.10-gentoo #4 SMP PREEMPT Thu Oct 31 21:23:56 CET 2013 x86_64 Intel(R) Core(TM) i7-2640M CPU @ 2.80GHz GenuineIntel GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.4/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/4.6.4/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-4.6.4/work/gcc-4.6.4/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.4 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.4/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.6.4 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.6.4/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/4.6.4/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.4/include/g++-v4 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/4.6.4/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 4.6.4 p1.2, pie-0.5.2' --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --disable-altivec --disable-fixed-point --enable-targets=all --disable-libgcj --enable-libgomp --disable-libmudflap --disable-libssp --enable-lto --without-cloog --without-ppl
Thread model: posix
gcc version 4.6.4 (Gentoo 4.6.4 p1.2, pie-0.5.2) 
****************************************************
Traceback (most recent call last):
  File "setup.py", line 6, in <module>
    from setuptools import setup, find_packages
ImportError: No module named setuptools

comment:7 follow-up: Changed 5 years ago by nbruin

Note that for pip to work, python must be built with OpenSSL support. Otherwise one gets

ImportError: cannot import name HTTPSHandler

So if we make pip a standard package, we would have to make OpenSSL a requirement for sage again (and/or use the openssl package for sage)

comment:8 in reply to: ↑ 6 Changed 5 years ago by tmonteil

Replying to jdemeyer:

pip-1.5.6 [...] ImportError?: No module named setuptools

Thanks for noticing, it worked for me and setuptools was not indicated as a dependency in SPKG.txt. I will fix this soon (but i am working on another branch from another version right now).

comment:9 in reply to: ↑ 7 ; follow-up: Changed 5 years ago by tmonteil

Replying to nbruin:

Note that for pip to work, python must be built with OpenSSL support. Otherwise one gets

ImportError: cannot import name HTTPSHandler

So if we make pip a standard package, we would have to make OpenSSL a requirement for sage again (and/or use the openssl package for sage)

This is a more serious issue, since IIRC, openssl can not be a standard package because of licensing stuff (which i do not understand well). Do you know if HTTPSHandler works with other SSL implementations (e.g. gnuTLS) ?

comment:10 in reply to: ↑ 9 ; follow-ups: Changed 5 years ago by nbruin

Replying to tmonteil:

This is a more serious issue, since IIRC, openssl can not be a standard package because of licensing stuff (which i do not understand well). Do you know if HTTPSHandler works with other SSL implementations (e.g. gnuTLS) ?

It depends on whether python's standard "Lib/ssl.py" module can be made to work with another implementation. As far as I can tell, Modules/_ssl.c is the real implementation underneath and that file looks like it's wired into openssl (it imports its headers unconditionally without alternatives listed). There may exist drop-in replacements? It's the kind of module you do not want to take any chances with because of security.

Solution I see:

  • pip is not essential for basic sage functionality, so while it's standard, its functioning can be conditional on recommended dependencies (rather than making them hard dependencies).
  • these dependencies can be met by a system-wide openssl install or by sage -i openssl; make

The big problem would probably be how we ship binaries, where "remaking" isn't an option. Can we just build WITH system openssl support and not include the shared library? Hopefully that would only lead to a fail at import ssl.

comment:11 Changed 5 years ago by git

  • Commit changed from 013e2e27978d4b48dd38ee3a00825f385286b0ba to 7380e00fcfecfc5d563b82ba5383cdea9b8f8ba7

Branch pushed to git repo; I updated commit sha1. New commits:

bf37567#17368 : add setuptools as a dependency.
7380e00#17368 add pip to COPYING.txt

comment:12 in reply to: ↑ 10 ; follow-up: Changed 5 years ago by tmonteil

  • Description modified (diff)
  • Status changed from needs_work to needs_review

Replying to nbruin:

The big problem would probably be how we ship binaries, where "remaking" isn't an option. Can we just build WITH system openssl support and not include the shared library? Hopefully that would only lead to a fail at import ssl.

If i understand correctly, official Sage binaries are built with the system's libssl-dev, so there should not be a problem for those (the user only have to install the libssl package from her distro to have it working).

It seems also that a similar dependency problem happens for ipython notebook: https://groups.google.com/forum/#!topic/sage-release/1teX_pCj8w0 http://article.gmane.org/gmane.comp.mathematics.sage.release/2310

comment:13 in reply to: ↑ 10 Changed 5 years ago by jdemeyer

Replying to nbruin:

  • pip is not essential for basic sage functionality, so while it's standard, its functioning can be conditional on recommended dependencies (rather than making them hard dependencies).

This is true for the moment, but we have to ensure that this remains true, that we never use pip to install a standard package.

comment:14 follow-up: Changed 5 years ago by novoselt

But part of the argument for having pip as standard is precisely to use it to install some of the packages, isn't it? Maybe openssl has to be made into a standard dependency. Both sagenb and ipython notebooks have issues without it already.

comment:15 in reply to: ↑ 14 Changed 5 years ago by nbruin

Replying to novoselt:

But part of the argument for having pip as standard is precisely to use it to install some of the packages, isn't it?

I'm not so sure it is. I certainly don't think it's desirable that building sage from a pre-downloaded tarball would need internet access to download further components. For one thing it would make it possible to get "failure to build sage due to inavailability of www.pypi.org", (or whatever error would result in if that were to happen).

comment:16 follow-up: Changed 5 years ago by novoselt

It has functionality to install pre-downloaded packages. Perhaps there will be no need for security library for this use.

comment:17 in reply to: ↑ 16 Changed 5 years ago by jdemeyer

Replying to novoselt:

It has functionality to install pre-downloaded packages.

On the other hand, for pre-downloaded Python packages, usually a simple python setup.py install works, so there is no real gain from using pip. For standard packages, we already have a build system which works.

I thought the main motivation for pip was to be able to easily install optional packages.

comment:18 in reply to: ↑ 12 ; follow-up: Changed 5 years ago by nbruin

Replying to tmonteil:

If i understand correctly, official Sage binaries are built with the system's libssl-dev, so there should not be a problem for those (the user only have to install the libssl package from her distro to have it working).

Yet it is surprisingly easy to miss that dependency. My system (fedora) ended up having a system python where openssl works, but sage-python builds without its support. I would not have minded if the build would have warned me at the start that openssl support isn't present and/or offered to download the openssl.spkg.

comment:19 in reply to: ↑ 18 Changed 5 years ago by jdemeyer

Replying to nbruin:

I would not have minded if the build would have warned me at the start that openssl support isn't present and/or offered to download the openssl.spkg.

I would mind since I don't care about OpenSSL.

Anyway, back to the ticket: I'm testing the build now and if that works, it can be set back to positive_review.

comment:20 Changed 5 years ago by jdemeyer

  • Reviewers changed from Andrey Novoseltsev to Andrey Novoseltsev, Jeroen Demeyer
  • Status changed from needs_review to positive_review

comment:21 Changed 5 years ago by vbraun

  • Branch changed from u/tmonteil/make_pip_a_standard_package to 7380e00fcfecfc5d563b82ba5383cdea9b8f8ba7
  • Resolution set to fixed
  • Status changed from positive_review to closed
Note: See TracTickets for help on using tickets.