Race condition when upgrading python

[embray]

I've encountered a race condition a couple times when doing a build on Cygwin where, for one reason or other, the python2 package gets upgraded (or python3 if --with-python=3). The error occurs in another package that is being built at the same time, I think when it tries to run the sage-flock command (which invokes /usr/bin/env python). That's also why I've only seen this on Cygwin--it's only on Cygwin that sage-flock is being invoked at the end of each package install.

Update: I have also now seen this on Linux too. It can also happen because sage-spkg-uninstall and sage-download-file use Python, among others. Any program use by sage-spkg that uses Python can break if it's called in the middle of upgrading or installing the Python in Sage.

The error ends up looking something like this:

[python2-2.7.15.p0] Copying package files from temporary location /home/embray/src/sagemath/sage/local/var/tmp/sage/build/python2-2.7.15.p0/inst to /home/embray/src/sagemath/sage/local
[ppl-1.2.p1] mkdir -p -- /home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/inst/home/embray/src/sagemath/sage/local/share/doc/ppl/ppl-user-c-interface-1.2-html
[ppl-1.2.p1] make[6]: Leaving directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src/doc'
[ppl-1.2.p1] make[5]: Leaving directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src/doc'
[ppl-1.2.p1] Making install in m4
[ppl-1.2.p1] make[5]: Entering directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src/m4'
[ppl-1.2.p1] make[6]: Entering directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src/m4'
[ppl-1.2.p1] make[6]: Nothing to be done for 'install-exec-am'.
[ppl-1.2.p1]  /usr/bin/mkdir -p '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/inst/home/embray/src/sagemath/sage/local/share/aclocal'
[ppl-1.2.p1]  /usr/bin/install -c -m 644 ppl.m4 ppl_c.m4 '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/inst/home/embray/src/sagemath/sage/local/share/aclocal'
[ppl-1.2.p1] make[6]: Leaving directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src/m4'
[ppl-1.2.p1] make[5]: Leaving directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src/m4'
[ppl-1.2.p1] make[4]: Leaving directory '/home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/src'
[ppl-1.2.p1]
[ppl-1.2.p1] real   14m49.948s
[ppl-1.2.p1] user   11m49.638s
[ppl-1.2.p1] sys    6m34.550s
[ppl-1.2.p1] Copying package files from temporary location /home/embray/src/sagemath/sage/local/var/tmp/sage/build/ppl-1.2.p1/inst to /home/embray/src/sagemath/sage/local
[ppl-1.2.p1] Could not find platform independent libraries <prefix>
[ppl-1.2.p1] Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
[ppl-1.2.p1] ImportError: No module named site
[ppl-1.2.p1] Waiting for rebase lock
[ppl-1.2.p1] Successfully installed ppl-1.2.p1

It says "Successfully installed" but then make exits with an error shortly thereafter.

The problem is that although ppl itself does not have a dependency on Python, sage-spkg itself does. But the python2 files are still in the process of being copied over. The bin/python has been copied, but not all of the stdlib, so that bin/python is broken.

I believe there are a few things that should be done about this. First of all, putting a lock around all modifications to $SAGE_LOCAL, as discussed in ​https://trac.sagemath.org/ticket/26018#comment:10 might help.

But I've mentioned in the context of other tickets that the build system itself should always stick to using the system python explicitly, and that Sage's Python should only be invoked explicitly in specific spkg-installs that need to build against the Sage Python.

[embray]

Opinions?

[jdemeyer]

Duplicate of #18438?

[embray]

I'll have to check out Volker's branch there. Something like sage-system-python is one possible solution I considered (on that I'm still not entirely happy with but it's better than nothing).

[vbraun]

I'm seeing this rather frequently now. The race was there for a long time but its getting more likely to hit.

[embray]

IF we're going to close this ticket (which is fine) the milestone / priority on the existing ticket should also be updated, and maybe link to this ticket which I believe adds more context.