#33078 closed enhancement (fixed)

Update package info for system python3 on Cygwin

Reported by: slelievre Owned by:
Priority: blocker Milestone: sage-9.6
Component: porting: Cygwin Keywords: cygwin, python3
Cc: mkoeppe, slelievre, gh-darijgr Merged in:
Authors: Matthias Koeppe Reviewers: Darij Grinberg
Report Upstream: Fixed upstream, in a later stable release. Work issues:
Branch: 65418de (Commits, GitHub, GitLab) Commit: 65418dec55872e7daa196cd13d2568ed66552b45
Dependencies: Stopgaps:

GitHub link to the corresponding issue

Description (last modified by gh-darijgr)

Sage 9.5.beta9 fails to use system Python 3.9 on Cygwin, unless configured with

./configure --with-python=/usr/bin/python3

Upstream report: https://cygwin.com/pipermail/cygwin/2021-December/250302.html

Change History (48)

comment:1 Changed 14 months ago by mkoeppe

logs please

comment:2 Changed 14 months ago by mkoeppe

Branch: u/mkoeppe/use_system_python_3_on_cygwin

comment:3 Changed 14 months ago by mkoeppe

Authors: Matthias Koeppe
Commit: 9cba975913c9ca3a4c7f7ff69a8a39dba80c04cd
Status: newneeds_review
Summary: Use system Python 3 on CygwinUpdate package info for system python3 on Cygwin

New commits:

9cba975build/pkgs/{_prereq,python3}/distros/cygwin.txt: Switch to python 3.9

comment:4 Changed 14 months ago by slelievre

Description: modified (diff)

Fails here. Something may be wrong with my Cygwin.

Before getting Python 3.9 there, Sage would build using system Python 3.8.12.

It now deems system Python 3.9 unusable, and builds a lacking Python, see #33079.

Logs for failed build of Sage 9.5.beta9 + #33078:

comment:5 Changed 14 months ago by mkoeppe

from config.log:

g++ -shared -Wl,--enable-auto-image-base conftest.dir/temp.cygwin-3.3.3-x86_64-3.9/conftest.o -L/home/s/sage-9-5-b9-33078/conftest_venv/lib/python3.9/config -L/usr/lib -lpython3.9 -o conftest.dir/lib.cygwin-3.3.3-x86_64-3.9/config_check_distutils_cxx.cpython-39-x86_64-cygwin.dll
	CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1"
configure:35820: result: no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://trac.sagemath.org/ticket/31132; to use it anyway, use ./configure --with-python=/usr/bin/python3

It's the -I/usr/include/uuid/ above that our configure script does not like.

./configure --with-python=/usr/bin/python3 as you wrote in the ticket description is a good workaround.

comment:6 Changed 14 months ago by mkoeppe

It might be worth reporting this to Cygwin as a bug

comment:7 Changed 14 months ago by mkoeppe

Report Upstream: N/ANot yet reported upstream; Will do shortly.

comment:8 Changed 14 months ago by mkoeppe

This is similar to the recent bug in Homebrew's packaging of python - #31132.

comment:9 Changed 14 months ago by mkoeppe

Description: modified (diff)
Report Upstream: Not yet reported upstream; Will do shortly.Reported upstream. No feedback yet.

comment:10 Changed 14 months ago by mkoeppe

Status: needs_reviewneeds_work

comment:12 Changed 14 months ago by mkoeppe

Cc: gh-darijgr added

comment:13 Changed 14 months ago by mkoeppe

Could someone who can run a local cygwin please test this branch with this cygwin package?

comment:14 Changed 14 months ago by gh-darijgr

What exactly should I test?

comment:15 Changed 14 months ago by mkoeppe

Install python39-3.9.9-3 and python39-devel-3.9.9-3 (these are test packages), then on this branch here run ./bootstrap && ./configure --with-system-python3=force && make build

Last edited 14 months ago by mkoeppe (previous) (diff)

comment:16 Changed 14 months ago by gh-darijgr

Hmm. Will this trigger recompilation of all other packages, or are these standalone?

comment:17 Changed 14 months ago by mkoeppe

Make that ./bootstrap && ./configure --with-system-python3=force --with-sage-venv && make build. If you are already on 9.5.beta9, then this will only rebuild Python packages in a new virtual environment.

comment:18 Changed 14 months ago by gh-darijgr

Sorry but I need more details. I have a freshly installed 9.5.beta9 with the regular python (and a couple patches for packages that would otherwise not have compiled). What should I type and in what order? And what should I do after that to get back to my previous install?

comment:19 Changed 14 months ago by mkoeppe

  1. In the cygwin terminal, type ./sage -python --version. This will probably output 3.8.x. Then type ls -l venv && ./config.status --config and save the output for later reference.
  1. Use the cygwin setup program. Search for python39, check "test" in the top right, and select python39-3.9.9-3 and python39-devel-3.9.9-3 for installation.
  1. In the cygwin terminal, type python3 --version; this should output 3.9.9
  1. In the cygwin terminal, type ./configure --with-system-python3=force and then verify that the output of ls -l venv includes python3.9.
  1. Type make build

To go back to the previous install, do ./configure --with-python=python3.8 --with-system-python3=force (adding options that were shown in step 0 by ./config.status --config)

comment:20 Changed 14 months ago by gh-darijgr

Thanks!

Hmm. There is no python39-devel-3.9.9-3 on the cygwin package list. Is that a problem?

skraeling@moria ~/sage
$ ./sage -python --version
Python 3.8.10

skraeling@moria ~/sage
$ ls -l venv && ./config.status --config
lrwxrwxrwx 1 skraeling None 33 Dec 28 18:03 venv -> local/var/lib/sage/venv-python3.8

comment:21 in reply to:  20 Changed 14 months ago by mkoeppe

Replying to gh-darijgr:

There is no python39-devel-3.9.9-3 on the cygwin package list.

There is but you need to enable "test packages" in the installer

comment:22 Changed 14 months ago by gh-darijgr

Here's what it shows: https://ibb.co/TKPSvFj Should I pick a better mirror?

comment:23 Changed 14 months ago by gh-darijgr

Also, the Python site https://www.python.org/downloads/release/python-393/ says:

NOTE: The release you're looking at has been recalled due to unintentional breakage of ABI compatibility with C extensions built in Python 3.9.0 - 3.9.2. Details in bpo-43710. Please use Python 3.9.4 or newer instead.

Wait for 3.9.4 to appear in Cygwin?

comment:24 Changed 14 months ago by mkoeppe

To right corner - check checkbox Test

comment:25 Changed 14 months ago by slelievre

We don't want 3.9.3, we want 3.9.9-3.

In your screenshot, the popup has "Skip", "3.9.9-1", "3.9.9-3 (Test)".

Pick "3.9.9-3 (Test)".

Same for python39-devel.

comment:26 Changed 14 months ago by slelievre

To clarify, Cygwin packages always add a "-1" etc "packaging version" to upstream version numbers.

For the Python 3.9.9 release, python39-3.9.9-1 was the first take at packaging it for Cygwin.

The new (test version) of a Cygwin package for the same Python 3.9.9 (with some cygwin-specific packaging fixes) is now python39-3.9.9-3, only available in Cygwin's "Test" packages until it has been tested enough.

comment:27 Changed 13 months ago by gh-darijgr

Ouch -- I need to get a better pair of glasses (or just stop multitasking).

I got python39-3.9.9-3 installed, but I don't see anything similar for python-devel on the cygwin package list. Where should I look for it?

comment:28 Changed 13 months ago by mkoeppe

python39-devel?

comment:29 Changed 13 months ago by gh-darijgr

The only option (even with Test checked) is 3.9.9-1. Could that be enough?

comment:30 Changed 13 months ago by mkoeppe

Should be fine.

comment:31 Changed 13 months ago by git

Commit: 9cba975913c9ca3a4c7f7ff69a8a39dba80c04cd65418dec55872e7daa196cd13d2568ed66552b45

Branch pushed to git repo; I updated commit sha1. New commits:

65418de.github/workflows/ci-cygwin*.yml: Use python39

comment:32 Changed 13 months ago by mkoeppe

Description: modified (diff)

comment:33 Changed 13 months ago by gh-darijgr

Description: modified (diff)

OK! I'm at the make build stage now. Hoping it doesn't again take 2 days...


New commits:

65418de.github/workflows/ci-cygwin*.yml: Use python39

comment:34 Changed 13 months ago by gh-darijgr

Confirming: mkoeppe's recipe (comment:19) works:

sage: python_help()

Welcome to Python 3.9's help utility!
[...]

I haven't tested the branch, though.

I assume I don't need to go back to the old version, seeing that it has worked?

comment:35 Changed 13 months ago by mkoeppe

Thanks for testing! I'll report to the cygwin maintainers that this version seems to be working well (and solves our problem).

comment:36 in reply to:  34 Changed 13 months ago by mkoeppe

Replying to gh-darijgr:

I assume I don't need to go back to the old version, seeing that it has worked?

I agree

comment:37 Changed 13 months ago by mkoeppe

Report Upstream: Reported upstream. No feedback yet.Fixed upstream, but not in a stable release.
Status: needs_workneeds_review

comment:38 Changed 13 months ago by mkoeppe

Let's wait for the fixed package to arrive in the main Cygwin repository - I assume there will be an announcement in https://cygwin.com/pipermail/cygwin/2022-January/thread.html

comment:39 Changed 13 months ago by mkoeppe

Milestone: sage-9.5sage-9.6

comment:40 Changed 13 months ago by mkoeppe

There is now python39-3.9.10-1, needs testing

comment:41 Changed 11 months ago by mkoeppe

Let's move this forward please

comment:42 Changed 11 months ago by gh-darijgr

Back to my office, so I'll start the install now. Expect an answer in 2 days or so (cygwin installs are slow even if lots of packages are preinstalled).

comment:43 Changed 11 months ago by gh-darijgr

real    2869m24.160s
user    651m22.088s
sys     542m26.524s
***************************************************************
Error building Sage.

The following package(s) may have failed to build (not necessarily
during this run of 'make all-build'):

* package:         scipy-1.7.3
  last build time: 2390901 Mar 24
  log file:        /home/dg899/sage/logs/pkgs/scipy-1.7.3.log

Here is the end of this log file:

  gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll
  /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: final link failed: Device or resource busy
  collect2: error: ld returned 1 exit status
  error: Command "gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll" failed with exit status 1

  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_ext.py

  ########### CLIB COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : gcc

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_clib.py
  error: subprocess-exited-with-error

  × Building wheel for scipy (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /home/dg899/sage/local/var/lib/sage/venv-python3.9/bin/python3 /home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpb06dquwe
  cwd: /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src
  Building wheel for scipy (pyproject.toml): finished with status 'error'
  ERROR: Failed building wheel for scipy
Failed to build scipy
ERROR: Failed to build one or more wheels

This looks like a parallel-compile-related heisenbug, so I'll now try "make build" again.

comment:44 Changed 11 months ago by gh-darijgr

As I hoped, it has now compiled without further problems and is working:

dg899@MATH-6264WD ~/sage
$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.6.beta5, Release Date: 2022-03-12               │
│ Using Python 3.9.10. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

comment:45 Changed 11 months ago by mkoeppe

Report Upstream: Fixed upstream, but not in a stable release.Fixed upstream, in a later stable release.

Thanks for testing!

comment:46 Changed 10 months ago by mkoeppe

Priority: majorcritical
Reviewers: Darij Grinberg
Status: needs_reviewpositive_review

I interpret this as a positive review

comment:47 Changed 10 months ago by mkoeppe

Priority: criticalblocker

comment:48 Changed 10 months ago by vbraun

Branch: u/mkoeppe/use_system_python_3_on_cygwin65418dec55872e7daa196cd13d2568ed66552b45
Resolution: fixed
Status: positive_reviewclosed
Note: See TracTickets for help on using tickets.