Install pip into py2 + py3 and fix py3 dependencies build

[vbraun]

Install both pip2 and pip3, otherwise your python2 + python3 isn't that useful.

Then, let sage-pip-install explicitly use pip2 or pip3.

With this ticket the SAGE_PYTHON3=yes build only fails in sagelib, but works for all dependencies.

This ticket also updates numpy to version 1.12.1.

[vbraun]

New commits:

​ed3f9c5

Install both pip2 and pip3, otherwise your python2 + python3 isn't that useful.

[chapoton]

+    echo "Skipping SageNB since it is not Python compatible"

this should say "not Python3 compatible"

[chapoton]

ok, I did the small change I suggested

---

New commits:

​df2c804	Merge branch 'u/vbraun/install_python_packages_into_2_and_3' in 8.0.b0
​742d9da	trac 22582 change error message for sagenb+python3

[vbraun]

Thanks, sounds good to me.

[chapoton]

This looks good to me. But I do not quite dare to set a positive review, because I have not tested that it does not break anything. Should I nevertheless set to positive, assuming that the buildbots will reveal any problem ?

[vbraun]

That would be the idea...

[chapoton]

ok, then let us try

[chapoton]

If I combine this ticket and its dependency, with SAGE_PYTHON3=yes, I get a failure on building cysignals:

[cysignals-1.3.2] config.status: executing build/src/cysignals/__init__.pxd commands
[cysignals-1.3.2] Traceback (most recent call last):
[cysignals-1.3.2]   File "setup.py", line 7, in <module>
[cysignals-1.3.2]     from Cython.Build.Dependencies import cythonize
[cysignals-1.3.2] ImportError: No module named Cython.Build.Dependencies
[cysignals-1.3.2] Error: could not determine package name
[cysignals-1.3.2] Error installing cysignals ... exiting

[vbraun]

This is because of

$ python setup.py --name
Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    from Cython.Build.Dependencies import cythonize

inside sage-pip-install. Cython is only installed into python3

[git]

Branch pushed to git repo; I updated commit sha1 and set ticket back to needs_review. New commits:

​c3850fa

Use the correct python in sage-pip-install

[chapoton]

If I try the latest branches for this ticket and its dependency with SAGE_PYTHON3=yes, it fails now on numpy:

[numpy-1.11.1.p0] Traceback (most recent call last):
[numpy-1.11.1.p0]   File "/home/chapoton/sage3/local/var/tmp/sage/build/numpy-1.11.1.p0/lapack_conf.py", line 3, in <module>
[numpy-1.11.1.p0]     import pkgconfig, os
[numpy-1.11.1.p0] ImportError: No module named pkgconfig

EDIT: This is preceded by

[pkgconfig-1.2.2.p0] Successfully installed pkgconfig-1.2.2.p0

[vbraun]

Works for me... complete logs?

[chapoton]

Exact same thing with "make build" and SAGE_PYTHON3=yes on another computer. I did not use "make dist-clean". Which logs do you wanna see? The numpy log:

Found local metadata for numpy-1.11.1.p0
Using cached file /home/chapoton/sage3/upstream/numpy-1.11.1.tar.gz
numpy-1.11.1.p0
====================================================
Setting up build directory for numpy-1.11.1.p0
Finished extraction
Applying patches from ../patches...
Applying ../patches/numpy-1.10.1-asarray_conversion.patch
patching file numpy/core/function_base.py
Applying ../patches/numpy-1.10.2-no-hardcode-blas.patch
patching file numpy/distutils/system_info.py
Hunk #1 succeeded at 1685 (offset -5 lines).
Hunk #2 succeeded at 1718 (offset -4 lines).
Applying ../patches/numpy-1.11.1-newlib-xlocale.patch
patching file numpy/core/setup_common.py
patching file numpy/core/src/multiarray/numpyos.c
****************************************************
Host system:
Linux icj-laptop 4.8.0-45-generic #48-Ubuntu SMP Fri Mar 24 11:46:39 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
****************************************************
C compiler: gcc
C compiler version:
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 6.2.0-5ubuntu12' --with-bugurl=file:///usr/share/doc/gcc-6/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-6 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-6-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-6-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-6-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 6.2.0 20161005 (Ubuntu 6.2.0-5ubuntu12) 
****************************************************
Traceback (most recent call last):
  File "/home/chapoton/sage3/local/var/tmp/sage/build/numpy-1.11.1.p0/lapack_conf.py", line 3, in <module>
    import pkgconfig, os
ImportError: No module named pkgconfig

real	0m0.021s
user	0m0.008s
sys	0m0.008s
************************************************************************
Error installing package numpy-1.11.1.p0
************************************************************************

[chapoton]

same thing after "make distclean"

[chapoton]

in fact, this already happens with just #22608 applied..

[vdelecroix]

Python 3 comes with pip, why do you want to install it!?

$ sage -sh -c "python3 -m pip"

What should be done is a proper shortcut pip3 -> python3 -m pip.

[jhpalmieri]

I tried to test this on OS X and ran into the problem that Sage's Python3 build is broken on OS X. See #22756.

[vdelecroix]

Replying to vdelecroix:

Python 3 comes with pip, why do you want to install it!?

$ sage -sh -c "python3 -m pip"

What should be done is a proper shortcut pip3 -> python3 -m pip.

ping?

[jhpalmieri]

I think the patch to the README.rst file in fpylll is not necessary in Sage 8.0.beta1: fpylll version 0.2.4dev does not have those non-ascii characters. (Or at least python3 does not complain about 0.2.4dev the way it does for 0.2.3dev.)

[jhpalmieri]

Replying to vdelecroix:

Python 3 comes with pip, why do you want to install it!?

$ sage -sh -c "python3 -m pip"

What should be done is a proper shortcut pip3 -> python3 -m pip.

If I just install Sage's python3 and its dependencies, then python3 -m pip tells me No module named pip, probably because I'm on OS X without openssl. That's an advantage to Sage's pip: it doesn't require Python's ssl module. So I say we should install it even with Python 3.

[jhpalmieri]

Replying to chapoton:

If I try the latest branches for this ticket and its dependency with SAGE_PYTHON3=yes, it fails now on numpy:

[numpy-1.11.1.p0] Traceback (most recent call last):
[numpy-1.11.1.p0]   File "/home/chapoton/sage3/local/var/tmp/sage/build/numpy-1.11.1.p0/lapack_conf.py", line 3, in <module>
[numpy-1.11.1.p0]     import pkgconfig, os
[numpy-1.11.1.p0] ImportError: No module named pkgconfig

EDIT: This is preceded by

[pkgconfig-1.2.2.p0] Successfully installed pkgconfig-1.2.2.p0

I see the same error. I think the problem is a race condition between python2 and python3: whichever one is built second creates a symlink local/bin/python, and that version of python is used to execute the script build/pkgs/numpy/lapack_conf.py. In my case, the link points to python2, which is not the right one if you have SAGE_PYTHON3=yes.

[vbraun]

pip2/3 has the right shebang pointing to the corresponding python

(sage-sh) vbraun@volker:git$ head -1 local/bin/pip?
==> local/bin/pip2 <==
#!/home/vbraun/Code/sage.git/local/bin/python2

==> local/bin/pip3 <==
#!/home/vbraun/Code/sage.git/local/bin/python3

So it is already a shortcut for corresponding-python-version -m pip

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​1d05b19	Merge tag '8.0.beta1' into t/22582/install_python_packages_into_2_and_3
​9e4522c	Make numpy build with py3 (includes update to 1.12.1)
​20340ea	Remove the fpylll README patch
​b7a4acb	Use correct python for creating the grammar pickle

[vbraun]

Fixed...

[jhpalmieri]

This doesn't fix the race condition, so sometimes python will point to python2, sometimes to python3. This will affect any packages that use python setup.py in their spkg-install, for example:

$ grep -R 'python setup' build/pkgs
./gambit/spkg-install:python setup.py --no-user-cfg build install
./numpy/spkg-install:sage-python setup.py \
./pillow/spkg-install:python setup.py \
./pycrypto/spkg-check:python setup.py test
./python_igraph/spkg-check:python setup.py check
./scons/spkg-install:python setup.py --no-user-cfg install

There are going to be other potential issues. See for example zn_poly's spkg-install:

# use 2to3 if we are running on python3
if python -c 'import sys; sys.exit(sys.version_info.major != 3)'; then
    2to3 -n -w --no-diffs makemakefile.py
fi

I think either we need to not create the symlink pointing to python3 in the python3 spkg-install and fix any resulting issues, or (my preference) we need to make sure the symlink points to the correct python by creating the appropriate symlink also in the python2 spkg-install, in case it gets built second.

Edit: run

grep -R -w python build/pkgs --include spkg-install

to see instances of python in spkg-install scripts. There will be more instances in spkg-check or other scripts, like matplotlib's make-setup-config.py. Having the symlink always point to python2 will require fixing all of these, and it will also make Sage harder to maintain.

[vbraun]

• Its easiest to just switch the python symlink at build time

vs

• pep 394 says: The more general python command should be installed whenever any version of Python 2 is installed and should invoke the same version of Python as the python2 command
• switching the python symlink introduces state (so you can't build other packages with and without SAGE_PYTHON3)
• distributions may or may not have python symlink to python2

I have a slight perference for the latter, tough I'm open for discussing this

Not all of the occurences of python in build/ are issues; Ideally such scripts are version-independent and work with either the system python or Sage's python2/3.

• matplotlib/make-setup-config.py needlessly depends on six and could easily be version-independent
• Calls to "python setup.py" should be replaced with sage-pip-install anyways

[jhpalmieri]

Should SAGE_PYTHON3 only have an effect at build time, or should it also control run time behavior? I guess it has to control run time behavior, at least as things are currently, many all of the relevant python packages get installed only to local/lib/python$VERSION/site-packages/. The symlink makes this easier (even trivial?) to implement.

I am not quite persuaded by PEP 394, since we're not really installing Python for general use; we're installing it for use within a specific environment, the Sage build environment.

Also, I agree that cleaning up things like python setup.py --> sage-pip-install would be great. For what it's worth, note that running matplotlib/make-setup-config.py with python3 led to the discovery of a minor bug -- see #22772.

[jhpalmieri]

Just to confirm that there are problems: I tested the current branch (+ #22756, #22765, #22772 so everything works on OS X), plus I added a dependency to make sure python3 got built before python2. Then with SAGE_PYTHON3=yes, pillow gets installed in python2.7/site-packages/, not in python3.5/site-packages/.

[jhpalmieri]

The numpy builds also fails with the error ImportError: No module named numpy, I think because of this in spkg-install:

# Touch all includes such that dependency checking works properly:
# the timestamp of the includes should be *now*, not the time when
# the numpy package was created.
python <<EOF
import os
os.chdir(os.environ["SAGE_ROOT"])  # Import numpy from safe location
import numpy
os.chdir(numpy.get_include())
os.system("touch numpy/*")
EOF

Easy enough to fix with your proposal: change python to sage_python.

[vbraun]

Its a bit premature to worry about how to run sage-on-python3, but one possibility would be to have a specific sage3 command. Using an environment variable doesn't sound clean to me.

[jhpalmieri]

I'm not sure which approach is the best; I'd like to hear other views. In any case, this needs work.

[jhpalmieri]

Replying to vbraun:

Its a bit premature to worry about how to run sage-on-python3, but one possibility would be to have a specific sage3 command. Using an environment variable doesn't sound clean to me.

Why not try to plan ahead?

As things stand, to run sage-on-python3, you would have to build with SAGE_PYTHON3=yes first, in which case sage-on-python2 will not work. So the build-time and run-time choices of Python version have to match up. I guess we could build everything for both Python 2 and 3, but that seems like a waste of space.

[jhpalmieri]

I also think we should merge the functools change soon, so in case this ticket takes a while, see #22770. Same for the sagenb change: #22787. Both of those are obstacles to building with SAGE_PYTHON3=yes right now, and I think are independent of this ticket.

If this ticket gets positively reviewed and merged soon, we can ignore the others, but in case this takes a while, we can at least fix those changes.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​ce87def	Make matplotlib/make-setup-config.py independent of six
​97c3fd8	Fix numpy spkg-install
​c6b7fae	Fix python -> python2 symlink
​6078c6b	Remove unneccesary python call from matplotlib spkg-install
​04fe607	Use sage-python to install pillow

[vbraun]

This now works for me (builds up to sagelib), with python -> python2 regardless of SAGE_PYTHON3, hopefully race-free.

The only thing that still ends up being installed with SAGE_PYTHON3=yes into lib/python2.7 is brial, which has a hardcoded python2.7 in its configure.ac

[vbraun]

Except that sagenb doesn't build on python2 since it trips over the /lib/python/ symlink missing

[jhpalmieri]

Does anyone else have any opinions on whether SAGE_PYTHON3=yes should lead to the creation of a symlink python -> python3?

[chapoton]

I have not followed in detail all the tickets around the one here. In my opinion, setting $SAGE_PYTHON3 to yes should do all what is needed at the build level so that, one day, when all our cython and python files compiles, sage will start with py3. I agree that the legacy sagenb can be desactivated when SAGE_PYTHON3 is yes.

[jhpalmieri]

I have a separate question for Volker: was the name build/bin/sage-python chosen intentionally to shadow the already existing local/bin/sage-python? That could lead to unintended (or intended?) consequences. Maybe it would be clearer to call it something like build/bin/sage-python2-or-3.

[vbraun]

IMHO we shouldn't tie our python3 support to where the python symlink points to. Pep 394 is pretty clear what it should be now, and it implies that it'll change in the future. Meaning that distributions will change over time; If we want Sage to be package-able then its just an additional unnecessary hurdle. Instead, do what everybody else does and have a sage3 launch script (or symlink to sage and look at argv[0]).

I didn't think about local/bin/sage-python, though I don't think there is a conflict. In fact, local/bin/sage-python seems to serve no useful purpose. Its only used from src/bin/sage-run to launch python from a shell script from python...

[jhpalmieri]

I agree that local/bin/sage-python serves no useful purpose. There is a conflict, though, because build/bin/sage-python will get run instead -- at least in my version of Sage, build/bin comes earlier in PATH than src/bin or local/bin -- and so the effect will be different depending on the setting of SAGE_PYTHON3.

[chapoton]

does not apply

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​10374c1

Merge tag '8.0.beta2' into t/22582/install_python_packages_into_2_and_3

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0e90d52	Trac 22764: remove the link SAGE_LOCAL/lib/python -> python2.7
​f33e86f	trac 22764: replace a few instances of
​b02ecf3	trac 22764: in a few comments and print statements, fix a few
​df6a763	trac 22764: rebase to Sage 8.0.beta1
​7cd0d62	trac 22764: merge with 8.0.beta1
​ab49413	trac 22764: change how check whether Sage is built.
​4715169	merging with 8.0.beta2
​fb90d5e	trac 22764: check whether Sage is built by looking for local/bin/sage
​dc46ea2	trac 22582: fix merge conflicts
​bc4c2f5	trac 22582: rename build/bin/sage-python to sage-python23

[jhpalmieri]

I fixed the merge problems. I also renamed build/bin/sage-python to build/bin/sage-python23 to avoid shadowing local/bin/sage-python -- the script in local/bin should not depend on the value of SAGE_PYTHON3, the way the script in build/bin should.

I am willing to give this a positive review, as long as this renaming is okay. A future ticket can deal with removing local/bin/sage-python.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​0d30e20

trac 22582: use sage-python23 to build the Sage library

[jhpalmieri]

Oops, one more change: we need to build the Sage library with sage-python23, not with python.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​7e7b5bc

trac 22582: build and test packages using sage-python23 instead of python

[jhpalmieri]

Some more changes: some spkg-check scripts were using python and should use sage-python23 instead: if we're building with it, we should test with it. Some optional packages also needed some modifications. I haven't tested these changes yet.

A few questions:

• when we build git, we pass the configure option --with-python="$SAGE_LOCAL"/bin/python. Does this matter? Does anyone use Sage's git as opposed to the system git?

• in the IPython spkg-install script, there is a test to see if python is version 3 or not:

  # add symlink if we are running on python3
  if python -c 'import sys; sys.exit(sys.version_info.major != 3)'; then
      cd "$SAGE_LOCAL"/bin
      rm -f ipython
      ln -s ipython3 ipython
  fi
  

  Keep this as is, or test instead whether SAGE_PYTHON3=yes, or delete the whole block?

[jhpalmieri]

I made another change in commit ​dc46ea2: we do not need to create a symlink local/bin/python pointing to python2: the python2 installation takes care of that automatically. The python3 installation does not create such a link, so we actually don't need to build python2 before python3. So we should remove python2 from the list of dependencies for python3.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​b954864

trac 22582: no need to make python2 a dependency for python3

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2dad104

trac 22582: a little documentation

[jhpalmieri]

Replying to vbraun:

The only thing that still ends up being installed with SAGE_PYTHON3=yes into lib/python2.7 is brial, which has a hardcoded python2.7 in its configure.ac

Should we open up a ticket for this?

[chapoton]

Replying to jhpalmieri:

Replying to vbraun:

The only thing that still ends up being installed with SAGE_PYTHON3=yes into lib/python2.7 is brial, which has a hardcoded python2.7 in its configure.ac

Should we open up a ticket for this?

Well, it seems that brial got installed in local/lib/python3.5/site-packages (during my latest build of sage with SAGE_PYTHON3=yes)

[jhpalmieri]

Replying to chapoton:

Replying to jhpalmieri:

Replying to vbraun:

The only thing that still ends up being installed with SAGE_PYTHON3=yes into lib/python2.7 is brial, which has a hardcoded python2.7 in its configure.ac

Should we open up a ticket for this?

Well, it seems that brial got installed in local/lib/python3.5/site-packages (during my latest build of sage with SAGE_PYTHON3=yes)

With no symlink local/lib/python pointing to local/lib/python3.5? As Volker points out, brial's file configure.ac has the line

AM_PATH_PYTHON([2.7])

For me, brial gets installed in local/lib/python2.7/site-packages, regardless of the variable SAGE_PYTHON3, if I'm building with this branch and the one from #22764.

[chapoton]

I am building with #22764 and #22756 (and other changes, including #22305 and #22775)

[jhpalmieri]

I'm trying again...

[jhpalmieri]

I still only see brial installed in local/lib/python2.7/.... I wonder what's going on.

[jhpalmieri]

Are you using a different version of brial?

[chapoton]

Just to be clear, I am not using the present branch (#22582) in my build.

[jhpalmieri]

My guess is that without using this branch, you have a link local/bin/python pointing to python3, so that is what gets used when installing brial.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​3f59504	trac 22582: compile brial using the appropriate version of Python.
​5a645fd	Merge branch 'public/22582' of trac.sagemath.org:sage into t/22582/public/22582

[jhpalmieri]

With this change, brial is built with the appropriate version of Python, as determined by SAGE_PYTHON3. In particular, it gets installed into local/lib/python3.5/site-packages if SAGE_PYTHON3=yes.

[jhpalmieri]

By the way, this issue with brial is an illustration of possible difficulties if we don't create a symlink python -> python3 when SAGE_PYTHON3=yes. I am willing to keep the symlink python -> python2, but it may create build problems which may be hard to debug.

[vbraun]

IMHO user issues will be even harder to debug if everybody has the python symlink pointing somewhere else, always depending on whether they originally (and potentially a long time ago) installed python with or without SAGE_PYTHON3.

Changes LGTM...

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​2179ac8

Merge branch 8.0.beta3 into t/22582/public/22582

[jhpalmieri]

Rebased. Anyone object to a positive review?

[dimpase]

Being just a bit more detailed in ticket description would have saved me the trouble of doing #22895... Oh well.

[dimpase]

What is the numpy tarball to be used here? It's not mentioned anywhere on the ticket. ​https://pypi.python.org/pypi/numpy only provides a link to zip file (that's what I used on #22895).

[vbraun]

The tarball for this ticket is on the mirrors. In general, tarballs for numpy can be found at ​https://github.com/numpy/numpy/releases

[gh-kliem]

(Google pointed me to this ticket, so I'm leaving this comment to help, in case someone encounters this.)

I ran into the same problem while updating to 9.0beta8 (including the update to python3).

Force installing python3 with

sage -f python3

(as pointed out above) solved it in my case. I did run make dist-clean and ./configure in case anyone wonders.

Replying to chapoton:

If I try the latest branches for this ticket and its dependency with SAGE_PYTHON3=yes, it fails now on numpy:

[numpy-1.11.1.p0] Traceback (most recent call last):
[numpy-1.11.1.p0]   File "/home/chapoton/sage3/local/var/tmp/sage/build/numpy-1.11.1.p0/lapack_conf.py", line 3, in <module>
[numpy-1.11.1.p0]     import pkgconfig, os
[numpy-1.11.1.p0] ImportError: No module named pkgconfig

EDIT: This is preceded by

[pkgconfig-1.2.2.p0] Successfully installed pkgconfig-1.2.2.p0