Race between building and using python

[vbraun]

I think what happened below is that the build system tried to use the sage-download-file script while Python was in the process of being installed. Rebuild works. We should probably use sage-native-execute python:

Found local metadata for ntl-6.2.1.p0
Attempting to download package ntl-6.2.1.p0
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
  File "/mnt/disk/home/buildslave-sage/slave/sage_git/build/src/bin/sage-download-file", line 11, in <module>
    import contextlib
  File "/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib/python2.7/contextlib.py", line 4, in <module>
    from functools import wraps
  File "/mnt/disk/home/buildslave-sage/slave/sage_git/build/local/lib/python2.7/functools.py", line 10, in <module>
    from _functools import partial, reduce
ImportError: No module named _functools
Error: spkg file '/mnt/disk/home/buildslave-sage/slave/sage_git/build/upstream/ntl-6.2.1.tar.bz2' not found.
This shouldn't happen, it is a bug in the sage-spkg script.

[vbraun]

New commits:

​d5bbfc0	Merge #18417 into #18428
​e280943	remove checksumming from sage-spkg
​aa24229	Do not ignor errors in sage-spkg
​49aa3c5	Only download mirror list when needed
​fd0b5ec	sage-spkg has to ignore some errors when building from scratch
​84c4620	Merge #18428 into #18438
​7f5bbf9	Always use the system python for sage-download-file

[vbraun]

sage-native-execute doesn't get rid of the whole Sage environment, we need something slightly different here that keeps SAGE_ROOT but resets PATH. This is now named sage-system-python

[nbruin]

There's way more necessary to use the system python! If you don't reset library paths and python search paths, you're setting yourself up for failure. #9386 is just some sh-style issues away from finalization and will provide proper environment reset tools, in a fairly efficient manner. Perhaps it's better to build on that? If you want to preserve SAGE_ROOT, you could remove it from the to-be-restored list, or set export SAGE_SAVED_SAGE_ROOT=$SAGE_ROOT (which means we should perhaps document a process to tweak restores).

[vbraun]

Well if #9386 can be finished then that would be great... but it didn't appear to be that close.

[nbruin]

From experience I know that if you are going to run a different python, you'll have to set PYTHONPATH and PYTHONHOME to the appropiate (previous) values too. I ran into this problem with a start-up script (in python) for magma. It didn't use anything fancy from Python, but at some point my system python and sage python (or their c-libs) were sufficiently different that the sage python libraries (picked up via PYTHONPATH, which was normally unset on my system) prevented python from running successfully.

[vbraun]

The system python ought to work without (=unset) PYTHONPATH/PYTHONHOME, which is what the sage-system-python script does. Of course in more complicated setups you'll need the right values...

[vbraun]

Here is another failure with more context:

Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_fnmatch.py ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_fork1.py ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_format.py ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_fpformat.py ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_fractions.pyFound local metadata for ntl-6.2.1.p0
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Traceback (most recent call last):
  File "/mnt/disk/home/release/Sage/src/bin/sage-download-file", line 11, in <module>
    import contextlib
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/contextlib.py", line 4, in <module>
    from functools import wraps
  File "/mnt/disk/home/release/Sage/local/lib/python2.7/functools.py", line 10, in <module>
    from _functools import partial, reduce
ImportError: No module named _functools
Makefile:1260: recipe for target '/mnt/disk/home/release/Sage/local/var/lib/sage/installed/ntl-6.2.1.p0' failed
make[2]: *** [/mnt/disk/home/release/Sage/local/var/lib/sage/installed/ntl-6.2.1.p0] Error 1
make[2]: *** Waiting for unfinished jobs....
 ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_frozen.py ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_ftplib.py ...
Compiling /mnt/disk/home/release/Sage/local/lib/python2.7/test/test_funcattrs.py ...

[embray]

See also #26083 for a similar report with some additional context. This has been becoming more common as we integrate more Python tools into sage-spkg. I feel that long term sage-spkg should be rewritten entirely in Python, but even that would not completely fix this problem. I agree with the general approach of preferring to use the system Python for the build system; otherwise trying to use tools in $SAGE_LOCAL to build $SAGE_LOCAL is destined to result in problems.

[embray]

The original branch for this doesn't totally make sense anymore, but I'm testing out a new implementation (which is actually much simpler now that Sage doesn't set PYTHONPATH, PYTHONHOME, LD_LIBRARY_PATH, etc.) But we also have more build scripts now that rely on Python so many of them also need to be updated.

[embray]

I've tested this a few times now on a 16 core machine with different variations of make -j from 8 to 16, so there should have been a high likelihood of reproducing the problem (as I confirmed against the unmodified 8.4.beta4), but with this patch I did not have any problems, as expected.

---

New commits:

​39fa363

Trac #18438: Always use system Python for build toolchain scripts written in

[embray]

Huh. I just tried this on a different system and ran into a teensy problem: It seems, for some reason, that sage-pip-install is now using the system pip rather than the one installed in Sage. Which is odd because I deliberately did not update sage-pip-install. It's supposed to use Sage's Python online.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​e6c18f1

Trac #18438: Ensure that sage-pip-install runs pip with Sage's Python

[embray]

I'm not wild about the workaround in sage-pip-install and am open to other ideas. On the other hand, maybe it's not all bad, as it really emphasizes and ensures that sage-pip-install is run with Sage's Python.

[git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

​ddafa36

Trac #18438: Always use system Python for build toolchain scripts written in

[embray]

(Decided to squash to a single commit)

[embray]

The sage-flock thing still causes things to break when installing cypari. Not sure why I never had this problem in earlier testing. Maybe I already had $SAGE_LOCAL on my $PATH to start with for some reason.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​d84bfb6

Trac #18438: A different take on sage-system-python

[embray]

I think this is a better middle-ground. It make sure to run the system Python, but otherwise keeps Sage's $PATH configuration.

If/when this has otherwise positive review I'll squash the commits again.

[dimpase]

looks good to me.