Opened 4 weeks ago

Last modified 8 days ago

#33078 needs_review enhancement

Update package info for system python3 on Cygwin

Reported by: slelievre Owned by:
Priority: major Milestone: sage-9.6
Component: porting: Cygwin Keywords: cygwin, python3
Cc: mkoeppe, slelievre, gh-darijgr Merged in:
Authors: Matthias Koeppe Reviewers:
Report Upstream: Fixed upstream, but not in a stable release. Work issues:
Branch: u/mkoeppe/use_system_python_3_on_cygwin (Commits, GitHub, GitLab) Commit: 65418dec55872e7daa196cd13d2568ed66552b45
Dependencies: Stopgaps:

Status badges

Description (last modified by gh-darijgr)

Sage 9.5.beta9 fails to use system Python 3.9 on Cygwin, unless configured with

./configure --with-python=/usr/bin/python3

Upstream report: https://cygwin.com/pipermail/cygwin/2021-December/250302.html

Change History (39)

comment:1 Changed 4 weeks ago by mkoeppe

logs please

comment:2 Changed 3 weeks ago by mkoeppe

  • Branch set to u/mkoeppe/use_system_python_3_on_cygwin

comment:3 Changed 3 weeks ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Commit set to 9cba975913c9ca3a4c7f7ff69a8a39dba80c04cd
  • Status changed from new to needs_review
  • Summary changed from Use system Python 3 on Cygwin to Update package info for system python3 on Cygwin

New commits:

9cba975build/pkgs/{_prereq,python3}/distros/cygwin.txt: Switch to python 3.9

comment:4 Changed 3 weeks ago by slelievre

  • Description modified (diff)

Fails here. Something may be wrong with my Cygwin.

Before getting Python 3.9 there, Sage would build using system Python 3.8.12.

It now deems system Python 3.9 unusable, and builds a lacking Python, see #33079.

Logs for failed build of Sage 9.5.beta9 + #33078:

comment:5 Changed 3 weeks ago by mkoeppe

from config.log:

g++ -shared -Wl,--enable-auto-image-base conftest.dir/temp.cygwin-3.3.3-x86_64-3.9/conftest.o -L/home/s/sage-9-5-b9-33078/conftest_venv/lib/python3.9/config -L/usr/lib -lpython3.9 -o conftest.dir/lib.cygwin-3.3.3-x86_64-3.9/config_check_distutils_cxx.cpython-39-x86_64-cygwin.dll
	CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1 -ggdb -O2 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fstack-protector-strong --param=ssp-buffer-size=4 -DOPENSSL_NO_SSL3=1 -I/usr/include/uuid/ -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/build=/usr/src/debug/python39-3.9.9-1 -fdebug-prefix-map=/pub/devel/python/python39/python39-3.9.9-1.x86_64/src/Python-3.9.9=/usr/src/debug/python39-3.9.9-1"
configure:35820: result: no, this is a misconfigured Python whose sysconfig compiler/linker flags contain -I or -L options, which may cause wrong versions of libraries to leak into the build of Python packages - see https://trac.sagemath.org/ticket/31132; to use it anyway, use ./configure --with-python=/usr/bin/python3

It's the -I/usr/include/uuid/ above that our configure script does not like.

./configure --with-python=/usr/bin/python3 as you wrote in the ticket description is a good workaround.

comment:6 Changed 3 weeks ago by mkoeppe

It might be worth reporting this to Cygwin as a bug

comment:7 Changed 3 weeks ago by mkoeppe

  • Report Upstream changed from N/A to Not yet reported upstream; Will do shortly.

comment:8 Changed 3 weeks ago by mkoeppe

This is similar to the recent bug in Homebrew's packaging of python - #31132.

comment:9 Changed 3 weeks ago by mkoeppe

  • Description modified (diff)
  • Report Upstream changed from Not yet reported upstream; Will do shortly. to Reported upstream. No feedback yet.

comment:10 Changed 3 weeks ago by mkoeppe

  • Status changed from needs_review to needs_work

comment:12 Changed 3 weeks ago by mkoeppe

  • Cc gh-darijgr added

comment:13 Changed 3 weeks ago by mkoeppe

Could someone who can run a local cygwin please test this branch with this cygwin package?

comment:14 Changed 3 weeks ago by gh-darijgr

What exactly should I test?

comment:15 Changed 3 weeks ago by mkoeppe

Install python39-3.9.9-3 and python39-devel-3.9.9-3 (these are test packages), then on this branch here run ./bootstrap && ./configure --with-system-python3=force && make build

Last edited 3 weeks ago by mkoeppe (previous) (diff)

comment:16 Changed 3 weeks ago by gh-darijgr

Hmm. Will this trigger recompilation of all other packages, or are these standalone?

comment:17 Changed 3 weeks ago by mkoeppe

Make that ./bootstrap && ./configure --with-system-python3=force --with-sage-venv && make build. If you are already on 9.5.beta9, then this will only rebuild Python packages in a new virtual environment.

comment:18 Changed 3 weeks ago by gh-darijgr

Sorry but I need more details. I have a freshly installed 9.5.beta9 with the regular python (and a couple patches for packages that would otherwise not have compiled). What should I type and in what order? And what should I do after that to get back to my previous install?

comment:19 Changed 3 weeks ago by mkoeppe

  1. In the cygwin terminal, type ./sage -python --version. This will probably output 3.8.x. Then type ls -l venv && ./config.status --config and save the output for later reference.
  1. Use the cygwin setup program. Search for python39, check "test" in the top right, and select python39-3.9.9-3 and python39-devel-3.9.9-3 for installation.
  1. In the cygwin terminal, type python3 --version; this should output 3.9.9
  1. In the cygwin terminal, type ./configure --with-system-python3=force and then verify that the output of ls -l venv includes python3.9.
  1. Type make build

To go back to the previous install, do ./configure --with-python=python3.8 --with-system-python3=force (adding options that were shown in step 0 by ./config.status --config)

comment:20 follow-up: Changed 3 weeks ago by gh-darijgr

Thanks!

Hmm. There is no python39-devel-3.9.9-3 on the cygwin package list. Is that a problem?

skraeling@moria ~/sage
$ ./sage -python --version
Python 3.8.10

skraeling@moria ~/sage
$ ls -l venv && ./config.status --config
lrwxrwxrwx 1 skraeling None 33 Dec 28 18:03 venv -> local/var/lib/sage/venv-python3.8

comment:21 in reply to: ↑ 20 Changed 3 weeks ago by mkoeppe

Replying to gh-darijgr:

There is no python39-devel-3.9.9-3 on the cygwin package list.

There is but you need to enable "test packages" in the installer

comment:22 Changed 3 weeks ago by gh-darijgr

Here's what it shows: https://ibb.co/TKPSvFj Should I pick a better mirror?

comment:23 Changed 3 weeks ago by gh-darijgr

Also, the Python site https://www.python.org/downloads/release/python-393/ says:

NOTE: The release you're looking at has been recalled due to unintentional breakage of ABI compatibility with C extensions built in Python 3.9.0 - 3.9.2. Details in bpo-43710. Please use Python 3.9.4 or newer instead.

Wait for 3.9.4 to appear in Cygwin?

comment:24 Changed 3 weeks ago by mkoeppe

To right corner - check checkbox Test

comment:25 Changed 3 weeks ago by slelievre

We don't want 3.9.3, we want 3.9.9-3.

In your screenshot, the popup has "Skip", "3.9.9-1", "3.9.9-3 (Test)".

Pick "3.9.9-3 (Test)".

Same for python39-devel.

comment:26 Changed 3 weeks ago by slelievre

To clarify, Cygwin packages always add a "-1" etc "packaging version" to upstream version numbers.

For the Python 3.9.9 release, python39-3.9.9-1 was the first take at packaging it for Cygwin.

The new (test version) of a Cygwin package for the same Python 3.9.9 (with some cygwin-specific packaging fixes) is now python39-3.9.9-3, only available in Cygwin's "Test" packages until it has been tested enough.

comment:27 Changed 3 weeks ago by gh-darijgr

Ouch -- I need to get a better pair of glasses (or just stop multitasking).

I got python39-3.9.9-3 installed, but I don't see anything similar for python-devel on the cygwin package list. Where should I look for it?

comment:28 Changed 3 weeks ago by mkoeppe

python39-devel?

comment:29 Changed 3 weeks ago by gh-darijgr

The only option (even with Test checked) is 3.9.9-1. Could that be enough?

comment:30 Changed 3 weeks ago by mkoeppe

Should be fine.

comment:31 Changed 3 weeks ago by git

  • Commit changed from 9cba975913c9ca3a4c7f7ff69a8a39dba80c04cd to 65418dec55872e7daa196cd13d2568ed66552b45

Branch pushed to git repo; I updated commit sha1. New commits:

65418de.github/workflows/ci-cygwin*.yml: Use python39

comment:32 Changed 3 weeks ago by mkoeppe

  • Description modified (diff)

comment:33 Changed 3 weeks ago by gh-darijgr

  • Description modified (diff)

OK! I'm at the make build stage now. Hoping it doesn't again take 2 days...


New commits:

65418de.github/workflows/ci-cygwin*.yml: Use python39

comment:34 follow-up: Changed 3 weeks ago by gh-darijgr

Confirming: mkoeppe's recipe (comment:19) works:

sage: python_help()

Welcome to Python 3.9's help utility!
[...]

I haven't tested the branch, though.

I assume I don't need to go back to the old version, seeing that it has worked?

comment:35 Changed 3 weeks ago by mkoeppe

Thanks for testing! I'll report to the cygwin maintainers that this version seems to be working well (and solves our problem).

comment:36 in reply to: ↑ 34 Changed 3 weeks ago by mkoeppe

Replying to gh-darijgr:

I assume I don't need to go back to the old version, seeing that it has worked?

I agree

comment:37 Changed 3 weeks ago by mkoeppe

  • Report Upstream changed from Reported upstream. No feedback yet. to Fixed upstream, but not in a stable release.
  • Status changed from needs_work to needs_review

comment:38 Changed 3 weeks ago by mkoeppe

Let's wait for the fixed package to arrive in the main Cygwin repository - I assume there will be an announcement in https://cygwin.com/pipermail/cygwin/2022-January/thread.html

comment:39 Changed 8 days ago by mkoeppe

  • Milestone changed from sage-9.5 to sage-9.6
Note: See TracTickets for help on using tickets.