Opened 17 months ago

Closed 16 months ago

Last modified 15 months ago

#29803 closed enhancement (fixed)

Upgrade setuptools, setuptools_scm, pip (2020-06), add package wheel; remove zope_interface

Reported by: mkoeppe Owned by:
Priority: major Milestone: sage-9.2
Component: packages: standard Keywords:
Cc: vbraun, chapoton, saraedum, slelievre, fbissey, dimpase, jhpalmieri Merged in:
Authors: Matthias Koeppe Reviewers: Dima Pasechnik
Report Upstream: N/A Work issues:
Branch: 371ede6 (Commits, GitHub, GitLab) Commit:
Dependencies: Stopgaps:

Status badges

Description (last modified by mkoeppe)

Big upgrade steps, enabled by drop python 2 support.

Add new packages to satisfy dependencies:

These are the latest versions as of 2020-06-05; all claim to support Python >= 3.5 (note #29033)

We adjust spkg-install of several packages by removing pyproject.toml to remove the need for more extensive build infrastructure. We disable the build of zope_interface (no longer needed, see #29754).

Tarballs: See checksums.ini [upstream_url]. (To configure Sage to download from the upstream URL, use ./configure --enable-download-from-upstream-url.)

Previous updates: #26969, #27886, #26008

Change History (38)

comment:1 Changed 17 months ago by mkoeppe

  • Branch set to u/mkoeppe/upgrade_setuptools__pip__2020_06_

comment:2 Changed 17 months ago by mkoeppe

  • Authors set to Matthias Koeppe
  • Cc slelievre fbissey added
  • Commit set to 50c696282b85af235f60cbdc6917bf7a2f9c6020
  • Description modified (diff)
  • Summary changed from Upgrade setuptools, pip (2020-06) to Upgrade setuptools, setuptools_scm, pip (2020-06)

New commits:

886693eUpgrade pip to 20.1.1
80374c5Upgrade setuptools to 47.1.1
50c6962Upgrade setuptools_scm to 4.1.2

comment:3 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:4 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:5 Changed 17 months ago by mkoeppe

pip installation fails, for example on ubuntu-bionic-standard (https://github.com/mkoeppe/sage/runs/741110905)

     Preparing wheel metadata: finished with status 'error'
ERROR: Command errored out with exit status 1: /sage/local/bin/python3 /sage/local/var/tmp/sage/build/pip-20.1.1/src/src/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp32gsdf31 Check the logs for full command output.

comment:6 Changed 17 months ago by mkoeppe

Seems we need to add the wheel package

comment:7 Changed 17 months ago by mkoeppe

  • Description modified (diff)
  • Summary changed from Upgrade setuptools, setuptools_scm, pip (2020-06) to Upgrade setuptools, setuptools_scm, pip (2020-06), add package wheel

comment:8 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:9 Changed 17 months ago by git

  • Commit changed from 50c696282b85af235f60cbdc6917bf7a2f9c6020 to 10a2bace146d47e45037a77e02da88f3bd4a22b0

Branch pushed to git repo; I updated commit sha1. New commits:

7c81b8cbuild/pkgs/wheel: New
70378edbuild/pkgs/pip/dependencies: Add wheel, simplify $(PYTHON) python3 to $(PYTHON)
10a2bacbuild/pkgs/setuptools, build/pkgs/pip: Remove python2 support from dependencies, install scripts

comment:11 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:12 Changed 17 months ago by mkoeppe

On debian-bullseye-standard:

  [pkgconfig-1.5.1]     File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
  [pkgconfig-1.5.1]       return _bootstrap._gcd_import(name[level:], package, level)
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  [pkgconfig-1.5.1]     File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
  [pkgconfig-1.5.1]   ModuleNotFoundError: No module named 'poetry'

comment:13 Changed 17 months ago by mkoeppe

pkgconfig is at the most current version. Looks like giving it a modern setuptools causes it to demand more.

comment:14 Changed 17 months ago by mkoeppe

  • Description modified (diff)

comment:15 Changed 17 months ago by mkoeppe

  • Description modified (diff)

poetry, of course, has a LONG list of additional dependencies... https://github.com/python-poetry/poetry/blob/master/pyproject.toml

comment:16 Changed 17 months ago by mkoeppe

  • Description modified (diff)

We will postpone adding poetry to #29810 and instead patch pkgconfig so it installs more prosaically using setuptools.

comment:18 Changed 17 months ago by mkoeppe

  • Cc dimpase added

See #28883 for the last pkgconfig update, which commented on the issues with poetry already.

comment:19 Changed 17 months ago by git

  • Commit changed from 10a2bace146d47e45037a77e02da88f3bd4a22b0 to 6755adb449240d6536a04c7c42b08042e0ed990f

Branch pushed to git repo; I updated commit sha1. New commits:

6755adbbuild/pkgs/pkgconfig/spkg-install.in: Remove pyproject.toml so that pip installs using setup.py

comment:21 Changed 17 months ago by mkoeppe

Several failures (for example in https://github.com/mkoeppe/sage/runs/744000563):

  [entrypoints-0.3]   ModuleNotFoundError: No module named 'flit'
  [testpath-0.4.2]   ModuleNotFoundError: No module named 'flit'
  [zope_interface-4.6.0]   ImportError: cannot import name 'Feature' from 'setuptools' (/sage/local/lib/python3.7/site-packages/setuptools/__init__.py)
  [terminado-0.8.1]   ModuleNotFoundError: No module named 'flit'

comment:22 Changed 17 months ago by mkoeppe

Possible that zope_interface can be removed (#29754)

comment:23 Changed 17 months ago by mkoeppe

  • Cc jhpalmieri added

comment:24 Changed 17 months ago by mkoeppe

entrypoints is current, https://pypi.org/project/testpath/ can be upgraded to 0.4.4, https://pypi.org/project/terminado/ to 0.8.3

comment:25 Changed 17 months ago by git

  • Commit changed from 6755adb449240d6536a04c7c42b08042e0ed990f to 5040d7ec7c28438800edcd84291413ae7f7dba9c

Branch pushed to git repo; I updated commit sha1. New commits:

29327d3build/pkgs/entrypoints: Do not provide setup.py, not needed for entrypoints 0.3
bcc79a5build/pkgs/entrypoints/spkg-install.in: Remove pyproject.toml so that setup.py is used
361b98bbuild/pkgs/terminado/spkg-install.in: Remove pyproject.toml so that setup.py is used
5040d7ebuild/pkgs/testpath/spkg-install.in: Remove pyproject.toml so that setup.py is used

comment:26 Changed 17 months ago by git

  • Commit changed from 5040d7ec7c28438800edcd84291413ae7f7dba9c to 371ede6fb66ab8dc10dec5a40ca14c425d7225a0

Branch pushed to git repo; I updated commit sha1. New commits:

371ede6build/pkgs/zope_interface: Change to a dummy script package

comment:27 Changed 17 months ago by mkoeppe

  • Description modified (diff)
  • Status changed from new to needs_review
  • Summary changed from Upgrade setuptools, setuptools_scm, pip (2020-06), add package wheel to Upgrade setuptools, setuptools_scm, pip (2020-06), add package wheel; remove zope_interface

comment:28 Changed 17 months ago by mkoeppe

Builds fine on homebrew-macos-standard. Ready for review.

comment:29 Changed 17 months ago by dimpase

  • Reviewers set to Dima Pasechnik
  • Status changed from needs_review to positive_review

lgtm

comment:30 Changed 17 months ago by mkoeppe

Thanks!

comment:31 Changed 16 months ago by vbraun

  • Branch changed from u/mkoeppe/upgrade_setuptools__pip__2020_06_ to 371ede6fb66ab8dc10dec5a40ca14c425d7225a0
  • Resolution set to fixed
  • Status changed from positive_review to closed

comment:32 Changed 16 months ago by jhpalmieri

  • Commit 371ede6fb66ab8dc10dec5a40ca14c425d7225a0 deleted

Not a big deal, but I'm curious: with pip, why only the change to set versions=3, instead of removing the loop for vers in $versions ... and replacing $vers with 3 in spkg-install.in?

comment:33 Changed 16 months ago by mkoeppe

I like to keep diffs minimal. Also minor uncertainty what the python binary will be called when python 4.0 comes out...

comment:34 Changed 15 months ago by jhpalmieri

I think that this ticket is leading to the following messages in the sagetex install log:

Successfully built sagetex
Installing collected packages: sagetex
  Created temporary directory: /private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q
  *** Error compiling '/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/extractsagecode.py'...
    File "/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/extractsagecode.py", line 48
      except getopt.GetoptError, err:
                               ^
  SyntaxError: invalid syntax
  
  *** Error compiling '/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/makestatic.py'...
    File "/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-unpacked-wheel-hw7nz29q/sagetex-3.4.data/data/share/texmf/tex/latex/sagetex/makestatic.py", line 48
      except getopt.GetoptError, err:
                               ^
  SyntaxError: invalid syntax
  

Successfully installed sagetex-3.4
Removed build tracker: '/private/var/folders/cp/n8wtqs490tq5psknff1hv9qr0000gn/T/pip-req-tracker-3d0gjhof'
Removing old SageTex version(s)

real	0m3.793s
user	0m1.162s
sys	0m0.458s
Copying package files from temporary location /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta3/local/var/tmp/sage/build/sagetex-3.4/inst to /Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta3/local
Successfully installed sagetex-3.4
Deleting temporary build directory
/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta3/local/var/tmp/sage/build/sagetex-3.4
Finished installing sagetex-3.4.spkg

Is it the new wheel package?

(This is on OS X using the system Python, version 3.7.7.)

comment:35 Changed 15 months ago by mkoeppe

pip likes to install via bdist_wheel now. Something is getting compiled that was not compiled before.

I think it reveals that the sagetex code is not compatible with python 3.

comment:36 follow-up: Changed 15 months ago by dimpase

I am going to make a new release of sagetex soon.

comment:37 in reply to: ↑ 36 Changed 15 months ago by jhpalmieri

Replying to dimpase:

I am going to make a new release of sagetex soon.

Good, that should take care of it. I'm surprised that sagetex still works, given these syntax errors.

comment:38 Changed 15 months ago by dimpase

probably it's from some file that we normally didn't install, or perhaps it was using some compatibility kludge that is now gone.

Note: See TracTickets for help on using tickets.