Opened 14 months ago
Closed 10 months ago
#33078 closed enhancement (fixed)
Update package info for system python3 on Cygwin
Reported by: | slelievre | Owned by: | |
---|---|---|---|
Priority: | blocker | Milestone: | sage-9.6 |
Component: | porting: Cygwin | Keywords: | cygwin, python3 |
Cc: | mkoeppe, slelievre, gh-darijgr | Merged in: | |
Authors: | Matthias Koeppe | Reviewers: | Darij Grinberg |
Report Upstream: | Fixed upstream, in a later stable release. | Work issues: | |
Branch: | 65418de (Commits, GitHub, GitLab) | Commit: | 65418dec55872e7daa196cd13d2568ed66552b45 |
Dependencies: | Stopgaps: |
Description (last modified by )
Sage 9.5.beta9 fails to use system Python 3.9 on Cygwin, unless configured with
./configure --with-python=/usr/bin/python3
Upstream report: https://cygwin.com/pipermail/cygwin/2021-December/250302.html
Change History (48)
comment:1 Changed 14 months ago by
comment:2 Changed 14 months ago by
Branch: | → u/mkoeppe/use_system_python_3_on_cygwin |
---|
comment:3 Changed 14 months ago by
Authors: | → Matthias Koeppe |
---|---|
Commit: | → 9cba975913c9ca3a4c7f7ff69a8a39dba80c04cd |
Status: | new → needs_review |
Summary: | Use system Python 3 on Cygwin → Update package info for system python3 on Cygwin |
New commits:
9cba975 | build/pkgs/{_prereq,python3}/distros/cygwin.txt: Switch to python 3.9
|
comment:4 Changed 14 months ago by
Description: | modified (diff) |
---|
comment:5 Changed 14 months ago by
from config.log:
g++ -shared -Wl,--enable-auto-image-base conftest.dir/temp.cygwin-3.3.3-x86_64-3.9/conftest.o -L/home/s/sage-9-5-b9-33078/conftest_venv/lib/python3.9/config -L/usr/lib -lpython3.9 -o conftest.dir/lib.cygwin-3.3.3-x86_64-3.9/config_check_distutils_cxx.cpython-39-x86_64-cygwin.dll CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1" configure:35820: result: no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://trac.sagemath.org/ticket/31132; to use it anyway, use ./configure --with-python=/usr/bin/python3
It's the -I/usr/include/uuid/
above that our configure script does not like.
./configure --with-python=/usr/bin/python3
as you wrote in the ticket description is a good workaround.
comment:7 Changed 14 months ago by
Report Upstream: | N/A → Not yet reported upstream; Will do shortly. |
---|
comment:8 Changed 14 months ago by
This is similar to the recent bug in Homebrew's packaging of python - #31132.
comment:9 Changed 14 months ago by
Description: | modified (diff) |
---|---|
Report Upstream: | Not yet reported upstream; Will do shortly. → Reported upstream. No feedback yet. |
comment:10 Changed 14 months ago by
Status: | needs_review → needs_work |
---|
comment:11 Changed 14 months ago by
New cygwin package version - https://cygwin.com/pipermail/cygwin/2021-December/250304.html
comment:12 Changed 14 months ago by
Cc: | gh-darijgr added |
---|
comment:13 Changed 14 months ago by
Could someone who can run a local cygwin please test this branch with this cygwin package?
comment:15 Changed 14 months ago by
Install python39-3.9.9-3
and python39-devel-3.9.9-3
(these are test packages),
then on this branch here run ./bootstrap && ./configure --with-system-python3=force && make build
comment:16 Changed 14 months ago by
Hmm. Will this trigger recompilation of all other packages, or are these standalone?
comment:17 Changed 14 months ago by
Make that ./bootstrap && ./configure --with-system-python3=force --with-sage-venv && make build
. If you are already on 9.5.beta9, then this will only rebuild Python packages in a new virtual environment.
comment:18 Changed 14 months ago by
Sorry but I need more details. I have a freshly installed 9.5.beta9 with the regular python (and a couple patches for packages that would otherwise not have compiled). What should I type and in what order? And what should I do after that to get back to my previous install?
comment:19 Changed 14 months ago by
- In the cygwin terminal, type
./sage -python --version
. This will probably output 3.8.x. Then typels -l venv && ./config.status --config
and save the output for later reference.
- Use the cygwin setup program. Search for
python39
, check "test" in the top right, and selectpython39-3.9.9-3
andpython39-devel-3.9.9-3
for installation.
- In the cygwin terminal, type
python3 --version
; this should output 3.9.9
- In the cygwin terminal, type
./configure --with-system-python3=force
and then verify that the output ofls -l venv
includespython3.9
.
- Type
make build
To go back to the previous install, do ./configure --with-python=python3.8 --with-system-python3=force
(adding options that were shown in step 0 by ./config.status --config
)
comment:20 follow-up: 21 Changed 14 months ago by
Thanks!
Hmm. There is no python39-devel-3.9.9-3
on the cygwin package list. Is that a problem?
skraeling@moria ~/sage $ ./sage -python --version Python 3.8.10 skraeling@moria ~/sage $ ls -l venv && ./config.status --config lrwxrwxrwx 1 skraeling None 33 Dec 28 18:03 venv -> local/var/lib/sage/venv-python3.8
comment:21 Changed 14 months ago by
Replying to gh-darijgr:
There is no
python39-devel-3.9.9-3
on the cygwin package list.
There is but you need to enable "test packages" in the installer
comment:22 Changed 14 months ago by
Here's what it shows: https://ibb.co/TKPSvFj Should I pick a better mirror?
comment:23 Changed 14 months ago by
Also, the Python site https://www.python.org/downloads/release/python-393/ says:
NOTE: The release you're looking at has been recalled due to unintentional breakage of ABI compatibility with C extensions built in Python 3.9.0 - 3.9.2. Details in bpo-43710. Please use Python 3.9.4 or newer instead.
Wait for 3.9.4 to appear in Cygwin?
comment:25 Changed 14 months ago by
We don't want 3.9.3, we want 3.9.9-3.
In your screenshot, the popup has "Skip", "3.9.9-1", "3.9.9-3 (Test)".
Pick "3.9.9-3 (Test)".
Same for python39-devel.
comment:26 Changed 14 months ago by
To clarify, Cygwin packages always add a "-1" etc "packaging version" to upstream version numbers.
For the Python 3.9.9 release, python39-3.9.9-1 was the first take at packaging it for Cygwin.
The new (test version) of a Cygwin package for the same Python 3.9.9 (with some cygwin-specific packaging fixes) is now python39-3.9.9-3, only available in Cygwin's "Test" packages until it has been tested enough.
comment:27 Changed 13 months ago by
Ouch -- I need to get a better pair of glasses (or just stop multitasking).
I got python39-3.9.9-3 installed, but I don't see anything similar for python-devel on the cygwin package list. Where should I look for it?
comment:29 Changed 13 months ago by
The only option (even with Test checked) is 3.9.9-1. Could that be enough?
comment:31 Changed 13 months ago by
Commit: | 9cba975913c9ca3a4c7f7ff69a8a39dba80c04cd → 65418dec55872e7daa196cd13d2568ed66552b45 |
---|
Branch pushed to git repo; I updated commit sha1. New commits:
65418de | .github/workflows/ci-cygwin*.yml: Use python39
|
comment:32 Changed 13 months ago by
Description: | modified (diff) |
---|
comment:33 Changed 13 months ago by
Description: | modified (diff) |
---|
OK! I'm at the make build
stage now. Hoping it doesn't again take 2 days...
New commits:
65418de | .github/workflows/ci-cygwin*.yml: Use python39
|
comment:34 follow-up: 36 Changed 13 months ago by
Confirming: mkoeppe's recipe (comment:19) works:
sage: python_help() Welcome to Python 3.9's help utility! [...]
I haven't tested the branch, though.
I assume I don't need to go back to the old version, seeing that it has worked?
comment:35 Changed 13 months ago by
Thanks for testing! I'll report to the cygwin maintainers that this version seems to be working well (and solves our problem).
comment:36 Changed 13 months ago by
Replying to gh-darijgr:
I assume I don't need to go back to the old version, seeing that it has worked?
I agree
comment:37 Changed 13 months ago by
Report Upstream: | Reported upstream. No feedback yet. → Fixed upstream, but not in a stable release. |
---|---|
Status: | needs_work → needs_review |
comment:38 Changed 13 months ago by
Let's wait for the fixed package to arrive in the main Cygwin repository - I assume there will be an announcement in https://cygwin.com/pipermail/cygwin/2022-January/thread.html
comment:39 Changed 13 months ago by
Milestone: | sage-9.5 → sage-9.6 |
---|
comment:42 Changed 11 months ago by
Back to my office, so I'll start the install now. Expect an answer in 2 days or so (cygwin installs are slow even if lots of packages are preinstalled).
comment:43 Changed 11 months ago by
real 2869m24.160s user 651m22.088s sys 542m26.524s *************************************************************** Error building Sage. The following package(s) may have failed to build (not necessarily during this run of 'make all-build'): * package: scipy-1.7.3 last build time: 2390901 Mar 24 log file: /home/dg899/sage/logs/pkgs/scipy-1.7.3.log
Here is the end of this log file:
gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll /usr/lib/gcc/x86_64-pc-cygwin/11/../../../../x86_64-pc-cygwin/bin/ld: final link failed: Device or resource busy collect2: error: ld returned 1 exit status error: Command "gcc -shared -Wl,--enable-auto-image-base -shared -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -L/home/dg899/sage/local/lib -Wl,-rpath,/home/dg899/sage/local/lib -D_GNU_SOURCE build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_misc.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/geom_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/global_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/io_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/libqhull_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/mem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/merge_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly2_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/poly_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/qset_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/random_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/rboxlib_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/stat_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/user_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/usermem_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_r.o build/temp.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull_src/src/userprintf_rbox_r.o -L/home/dg899/sage/local/lib -L/home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/config -L/usr/lib -Lbuild/temp.cygwin-3.3.4-x86_64-3.9 -lopenblas.dll -lopenblas.dll -lpython3.9 -o build/lib.cygwin-3.3.4-x86_64-3.9/scipy/spatial/qhull.cpython-39-x86_64-cygwin.dll" failed with exit status 1 ########### EXT COMPILER OPTIMIZATION ########### Platform : Architecture: x64 Compiler : gcc CPU baseline : Requested : 'min' Enabled : SSE SSE2 SSE3 Flags : -msse -msse2 -msse3 Extra checks: none CPU dispatch : Requested : 'max -xop -fma4' Enabled : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL Generated : none CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_ext.py ########### CLIB COMPILER OPTIMIZATION ########### Platform : Architecture: x64 Compiler : gcc CPU baseline : Requested : 'min' Enabled : SSE SSE2 SSE3 Flags : -msse -msse2 -msse3 Extra checks: none CPU dispatch : Requested : 'max -xop -fma4' Enabled : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_KNM AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL Generated : none CCompilerOpt.cache_flush[809] : write cache to path -> /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src/build/temp.cygwin-3.3.4-x86_64-3.9/ccompiler_opt_cache_clib.py error: subprocess-exited-with-error × Building wheel for scipy (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. full command: /home/dg899/sage/local/var/lib/sage/venv-python3.9/bin/python3 /home/dg899/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /tmp/tmpb06dquwe cwd: /home/dg899/sage/local/var/lib/sage/venv-python3.9/var/tmp/sage/build/scipy-1.7.3/src Building wheel for scipy (pyproject.toml): finished with status 'error' ERROR: Failed building wheel for scipy Failed to build scipy ERROR: Failed to build one or more wheels
This looks like a parallel-compile-related heisenbug, so I'll now try "make build" again.
comment:44 Changed 11 months ago by
As I hoped, it has now compiled without further problems and is working:
dg899@MATH-6264WD ~/sage $ ./sage ┌────────────────────────────────────────────────────────────────────┐ │ SageMath version 9.6.beta5, Release Date: 2022-03-12 │ │ Using Python 3.9.10. Type "help()" for help. │ └────────────────────────────────────────────────────────────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Warning: this is a prerelease version, and it may be unstable. ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
comment:45 Changed 11 months ago by
Report Upstream: | Fixed upstream, but not in a stable release. → Fixed upstream, in a later stable release. |
---|
Thanks for testing!
comment:46 Changed 10 months ago by
Priority: | major → critical |
---|---|
Reviewers: | → Darij Grinberg |
Status: | needs_review → positive_review |
I interpret this as a positive review
comment:47 Changed 10 months ago by
Priority: | critical → blocker |
---|
comment:48 Changed 10 months ago by
Branch: | u/mkoeppe/use_system_python_3_on_cygwin → 65418dec55872e7daa196cd13d2568ed66552b45 |
---|---|
Resolution: | → fixed |
Status: | positive_review → closed |
logs please