Update package info for system python3 on Cygwin

[slelievre]

Sage 9.5.beta9 fails to use system Python 3.9 on Cygwin, unless configured with

./configure --with-python=/usr/bin/python3

Upstream report: ​https://cygwin.com/pipermail/cygwin/2021-December/250302.html

[mkoeppe]

logs please

[mkoeppe]

New commits:

​9cba975

build/pkgs/{_prereq,python3}/distros/cygwin.txt: Switch to python 3.9

[slelievre]

Fails here. Something may be wrong with my Cygwin.

Before getting Python 3.9 there, Sage would build using system Python 3.8.12.

It now deems system Python 3.9 unusable, and builds a lacking Python, see #33079.

Logs for failed build of Sage 9.5.beta9 + #33078:

• ​https://aij.ovh/sage/logs/slel-2021-12-27-cyg-sage-9-5-b9-33078.zip

[mkoeppe]

from config.log:

g++ -shared -Wl,--enable-auto-image-base conftest.dir/temp.cygwin-3.3.3-x86_64-3.9/conftest.o -L/home/s/sage-9-5-b9-33078/conftest_venv/lib/python3.9/config -L/usr/lib -lpython3.9 -o conftest.dir/lib.cygwin-3.3.3-x86_64-3.9/config_check_distutils_cxx.cpython-39-x86_64-cygwin.dll
	CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1"
configure:35820: result: no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://trac.sagemath.org/ticket/31132; to use it anyway, use ./configure --with-python=/usr/bin/python3

It's the -I/usr/include/uuid/ above that our configure script does not like.

./configure --with-python=/usr/bin/python3 as you wrote in the ticket description is a good workaround.

[mkoeppe]

It might be worth reporting this to Cygwin as a bug

[mkoeppe]

This is similar to the recent bug in Homebrew's packaging of python - #31132.

[mkoeppe]

New cygwin package version - ​https://cygwin.com/pipermail/cygwin/2021-December/250304.html

[mkoeppe]

Could someone who can run a local cygwin please test this branch with this cygwin package?

[gh-darijgr]

What exactly should I test?

[mkoeppe]

Install python39-3.9.9-3 and python39-devel-3.9.9-3 (these are test packages), then on this branch here run ./bootstrap && ./configure --with-system-python3=force && make build

[gh-darijgr]

Hmm. Will this trigger recompilation of all other packages, or are these standalone?

[mkoeppe]

Make that ./bootstrap && ./configure --with-system-python3=force --with-sage-venv && make build. If you are already on 9.5.beta9, then this will only rebuild Python packages in a new virtual environment.

[gh-darijgr]

Sorry but I need more details. I have a freshly installed 9.5.beta9 with the regular python (and a couple patches for packages that would otherwise not have compiled). What should I type and in what order? And what should I do after that to get back to my previous install?

[mkoeppe]

0. In the cygwin terminal, type ./sage -python --version. This will probably output 3.8.x. Then type ls -l venv && ./config.status --config and save the output for later reference.

1. Use the cygwin setup program. Search for python39, check "test" in the top right, and select python39-3.9.9-3 and python39-devel-3.9.9-3 for installation.

2. In the cygwin terminal, type python3 --version; this should output 3.9.9

3. In the cygwin terminal, type ./configure --with-system-python3=force and then verify that the output of ls -l venv includes python3.9.

4. Type make build

To go back to the previous install, do ./configure --with-python=python3.8 --with-system-python3=force (adding options that were shown in step 0 by ./config.status --config)

[gh-darijgr]

Thanks!

Hmm. There is no python39-devel-3.9.9-3 on the cygwin package list. Is that a problem?

skraeling@moria ~/sage
$ ./sage -python --version
Python 3.8.10

skraeling@moria ~/sage
$ ls -l venv && ./config.status --config
lrwxrwxrwx 1 skraeling None 33 Dec 28 18:03 venv -> local/var/lib/sage/venv-python3.8

[mkoeppe]

Replying to gh-darijgr:

There is no python39-devel-3.9.9-3 on the cygwin package list.

There is but you need to enable "test packages" in the installer

[gh-darijgr]

Here's what it shows: ​https://ibb.co/TKPSvFj Should I pick a better mirror?

[gh-darijgr]

Also, the Python site ​https://www.python.org/downloads/release/python-393/ says:

NOTE: The release you're looking at has been recalled due to unintentional breakage of ABI compatibility with C extensions built in Python 3.9.0 - 3.9.2. Details in bpo-43710. Please use Python 3.9.4 or newer instead.

Wait for 3.9.4 to appear in Cygwin?

[mkoeppe]

To right corner - check checkbox Test

[slelievre]

We don't want 3.9.3, we want 3.9.9-3.

In your screenshot, the popup has "Skip", "3.9.9-1", "3.9.9-3 (Test)".

Pick "3.9.9-3 (Test)".

Same for python39-devel.

[slelievre]

To clarify, Cygwin packages always add a "-1" etc "packaging version" to upstream version numbers.

For the Python 3.9.9 release, python39-3.9.9-1 was the first take at packaging it for Cygwin.

The new (test version) of a Cygwin package for the same Python 3.9.9 (with some cygwin-specific packaging fixes) is now python39-3.9.9-3, only available in Cygwin's "Test" packages until it has been tested enough.

[gh-darijgr]

Ouch -- I need to get a better pair of glasses (or just stop multitasking).

I got python39-3.9.9-3 installed, but I don't see anything similar for python-devel on the cygwin package list. Where should I look for it?

[mkoeppe]

python39-devel?

[gh-darijgr]

The only option (even with Test checked) is 3.9.9-1. Could that be enough?

[mkoeppe]

Should be fine.

[git]

Branch pushed to git repo; I updated commit sha1. New commits:

​65418de

.github/workflows/ci-cygwin*.yml: Use python39

[gh-darijgr]

OK! I'm at the make build stage now. Hoping it doesn't again take 2 days...

---

New commits:

​65418de

.github/workflows/ci-cygwin*.yml: Use python39

[gh-darijgr]

Confirming: mkoeppe's recipe (comment:19) works:

sage: python_help()

Welcome to Python 3.9's help utility!
[...]

I haven't tested the branch, though.

I assume I don't need to go back to the old version, seeing that it has worked?

[mkoeppe]

Thanks for testing! I'll report to the cygwin maintainers that this version seems to be working well (and solves our problem).

[mkoeppe]

Replying to gh-darijgr:

I assume I don't need to go back to the old version, seeing that it has worked?

I agree

[mkoeppe]

Let's wait for the fixed package to arrive in the main Cygwin repository - I assume there will be an announcement in ​https://cygwin.com/pipermail/cygwin/2022-January/thread.html

[mkoeppe]

There is now python39-3.9.10-1, needs testing

[mkoeppe]

Let's move this forward please

[gh-darijgr]

Back to my office, so I'll start the install now. Expect an answer in 2 days or so (cygwin installs are slow even if lots of packages are preinstalled).

[gh-darijgr]

real    2869m24.160s
user    651m22.088s
sys     542m26.524s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make all-build'):

* package:         scipy-1.7.3
  last build time: 2390901 Mar 24
  log file:        /home/dg899/sage/logs/pkgs/scipy-1.7.3.log

Here is the end of this log file:

  gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll
  /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: final link failed: Device or resource busy
  collect2: error: ld returned 1 exit status
  error: Command "gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll" failed with exit status 1

  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_ext.py

  ########### CLIB COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_clib.py
  error: subprocess-exited-with-error

  × Building wheel for scipy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/dg899/sage/local/var/lib/sage/venv-python3.9/bin/python3 /home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpb06dquwe
  cwd: /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src
  Building wheel for scipy (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for scipy
Failed to build scipy
ERROR: Failed to build one or more wheels

This looks like a parallel-compile-related heisenbug, so I'll now try "make build" again.

[gh-darijgr]

As I hoped, it has now compiled without further problems and is working:

dg899@MATH-6264WD ~/sage
$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.6.beta5, Release Date: 2022-03-12               │
│ Using Python 3.9.10. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

[mkoeppe]

Thanks for testing!

[mkoeppe]

I interpret this as a positive review